Sensor calibration with environment map

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sensor calibration with environment map. In some implementations, a three-dimensional surfel representation of a real-world environment is obtained. One or more surfels of the surfel representation having a particular classification of the different classifications are selected. Input sensor data from one or more sensors installed on an autonomous or semi-autonomous vehicle are received. The input sensor data is compared to the surfel representation to identify one or more differences between the observation and the surfel representation. At least one sensor of the one or more sensors is calibrated using the one or more differences between the observation and the surfel representation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority of U.S. patent application Ser. No.17/139,557 filed Dec. 31, 2020. The content of the prior application isincorporated herein by reference in its entirety.

TECHNICAL FIELD

This specification relates to techniques for calibrating sensors, andfor navigating autonomous or semi-autonomous vehicles.

BACKGROUND

Autonomous vehicles include self-driving cars, boats, and aircraft.Autonomous vehicles use a variety of on-board sensors in tandem with maprepresentations of the environment in order to make control andnavigation decisions.

Some vehicles use maps to navigate or otherwise operate in theenvironment, where the maps include representations of characteristicsof the operating environment. However, these maps may includerepresentations of characteristics in the environment that areunreliable, such as those representations of characteristics in theenvironment that change over time or that are likely to change overtime. As such, the operation of these vehicles may, in part, rely onrepresentations in the maps that are unreliable, which can lead toerrors or inefficiencies during the navigation of these vehicles in theenvironment.

Accordingly, one challenging aspect of autonomous vehicle operation isidentifying and accounting for the inherently unreliable characteristicsof an environment that can change over time. Thus, a common problem isthe navigation systems of vehicles becoming confused as to where it isspecifically located with respect to environment, and/or makingnavigation errors when a current view of the environment does not matcha corresponding map of the environment due to changes in the environmentfrom a time that the map is generated and a present time. Theseunreliable representations can also lead to other problems, such assensors of the navigation systems being miscalibrated.

SUMMARY

This specification describes how a vehicle, e.g., an autonomous orsemi-autonomous vehicle, can use a surfel map that represents areal-world environment to modify a surfel map that represents areal-world environment to improve calibration of on-board sensors.Notably, a subset of surfels in the surfel map may be selected to beused for the calibration of one or more of the vehicle's on-boardsensors. This subset of surfels may be identified based on a determinedor predicted reliability of the surfels in representing the real-worldenvironment. For example, the subset of surfels may include surfels thatrepresent objects in the real-world environment are determined withsufficient likelihood not to change position and/or appearance. In usingthis subset of surfels for calibration, the operation of the vehicle inthe environment can be improved, by, for example, improving the accuracyof the perceived location of the vehicle in the environment, improvingthe accuracy of perceived locations of objects in the environment, andimproving the calibration accuracy of various sensors of a navigationsystem of the vehicle. As another advantage, by focusing calibrationefforts on only the subset of surfels in the surfel map, the amount ofsensor data required to be collected and, relatedly, the processingrequirements on the vehicle's navigation system can be reduced.

In some implementations, the surfel map is modified to include only thesubset of reliable surfels. For example, the surfel map may be modifiedto have all other surfels removed or hidden from the surfel map. Havingbeen simplified with respect to the surfel map, the resulting modifiedsurfel map can reduce the amount of sensor data required to be collectedand the processing requirements on the vehicle's navigation system toanalyze the modified surfel map and process the sensor data.

As an example, the navigation system of the vehicle may access a surfelmap corresponding to a particular section of a road that the vehicle iscurrently traveling along. The navigation system may proceed to hide allsurfels in the surfel map that are associated with a type of object thatis deemed unreliable. For example, the navigation system may hide anysurfels that have been tagged as representing a plant, a person, ananimal, or a vehicle. The navigation system may use the resultingmodified surfel map to determine what sensor data to collect, and/or asubset of collected sensor data that is required to be analyzed. Forexample, the navigation system may only collect sensor data in adirection that corresponds to one or more objects having representationsin the modified surfel map. Similarly, the navigation system may onlyprocess sensor data corresponding to one or more objects havingrepresentations in the modified surfel map.

In some implementations, the vehicle, such as a navigation system of thevehicle, uses the modified surfel map to calibrate one or more sensorsof the navigation system. For example, the navigation system may obtainsensor data corresponding to a reliable object located in theenvironment, the reliable object having a representation in the modifiedsurfel map. The navigation system may analyze the sensor data toidentify a corresponding area in the modified surfel map. The navigationsystem may proceed to compare the sensor data to the corresponding area,and based on differences between the sensor data and the area of themodified surfel map, update the parameters of the one or more sensors.

As an example, the navigation system may use an onboard camera tocollect image data and a lidar sensor to collect laser data. Thenavigation system may analyze the image data to identify features in theimage data, and proceed to match those features to correspondingfeatures in the modified surfel map. By comparing the features in theimages to corresponding features in the modified surfel map, thenavigation system can determine changes to the position and/ororientation of the camera (and/or the lidar sensor) needed to align thecamera (and/or the lidar sensor) to the modified surfel map. Thenavigation system can use these changes to the position and/ororientation for alignment to modify the collected laser data to alignthe laser data with the modified surfel map. The navigation system canthen perform point matching between the laser data and the modifiedsurfel map to identify, for example, differences between observed pointdistances and expected point distances, and differences between observedbeam angles and expected beam angles. The results can be used to updatevarious parameters of the camera and the lidar sensor. For example, anoffset range measurement can be updated. Moreover, roll, pitch, and yawangles between the camera and the lidar sensor can be updated.

In one general aspect, a method includes: obtaining a three-dimensionalsurfel representation of a real-world environment, the surfelrepresentation includes a plurality of surfels, where each of thesurfels corresponds to a respective location of plurality of locationsin a three-dimensional space of the real-world environment, and wheremultiple surfels of the plurality of surfels are associated with aplurality of different classifications; selecting one or more surfels ofthe surfel representation having a particular classification f thedifferent classifications; receiving input sensor data from one or moresensors installed on an autonomous or semi-autonomous vehicle, the inputsensor data indicating an observation of at least a portion of thereal-world environment; comparing the input sensor data to the surfelrepresentation to identify one or more differences between theobservation and the surfel representation; and calibrating at least onesensor of the one or more sensors using the one or more differencesbetween the observation and the surfel representation.

Implementations include one or more of the following features. Forexample, in some implementations, the method includes updating thesurfel representation based on the one or more surfels, where comparingthe input sensor data to the surfel representation to identify the oneor more differences between the observation and the surfelrepresentation includes comparing the input sensor data to the updatedsurfel representation to identify one or more differences between theobservation and the updated surfel representation.

In some implementations, updating the surfel representation based on theone or more surfels includes filtering the surfel representation toremove or hide the one or more surfels having the particularclassification.

In some implementations, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingall surfels in the surfel representation that have the particularclassification; and updating the surfel representation based on the oneor more surfels includes filtering the surfel representation to removeor hide all surfels in the representation that have the particularclassification.

In some implementations, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingone or more surfels in the surfel representation that have beenassociated with a particular object of the real-world environment; andupdating the surfel representation based on the one or more surfelsincludes filtering the surfel representation to remove or hide allsurfels in the surfel representation that have been associated with aparticular object of the real-world environment.

In some implementations, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingthe one or more surfels in the surfel representation that have theparticular classification based on a type of environment of thereal-world environment; and updating the surfel representation based onthe one or more surfels includes filtering the surfel representation toremove or hide the one or more surfels.

In some implementations, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingone or more surfels in the surfel representation that have beenidentified as untrustworthy; and updating the surfel representationbased on the one or more surfels includes filtering the surfelrepresentation to remove or hide the one or more surfels in the surfelrepresentation that have been identified as untrustworthy.

In some implementations, updating the surfel representation based on theone or more surfels includes attaching a tag to surfels in the surfelrepresentation other than the one or more surfels, where the tagindicates that the corresponding surfels are sufficiently reliable.

In some implementations, updating the surfel representation based on theone or more surfels includes attaching a tag to the one or more surfels,where the tag indicates that the corresponding surfels are notsufficiently reliable.

In some implementations, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingall surfels in the surfel representation that are associated withvegetation.

In some implementations, selecting the one more surfels of the surfelrepresentation having the particular classification includes selectingall surfels in the surfel representation that are associated with typesof objects other than one or more of the following types of objects:buildings; roadways; road signs; or road barriers.

In some implementations, receiving input sensor data from the sensorsinstalled on the autonomous or semi-autonomous vehicle includesreceiving laser data from a lidar system, and the one or more sensorsinstalled on the autonomous or semi-autonomous vehicle include the lidarsystem.

In some implementations, comparing the input sensor data to the surfelrepresentation to identify the one or more differences between theobservation and the surfel representation includes: estimatingthree-dimensional point data from the laser data, where the observationis the three-dimensional point data; and performing dense image matchingusing the three-dimensional point data to identify point matches betweenthe observation and the surfel representation, the one or moredifferences between the observation and the surfel representationinclude one or more distances between the matching points in the pointdata and corresponding surfels in the surfel representation, andcalibrating the at least one sensor includes calibrating the lidarsystem using the one or more distances.

In some implementations, receiving input sensor data from the one ormore sensors installed on the autonomous or semi-autonomous vehicleincludes receiving image data from one or more cameras, and the one ormore sensors installed on the autonomous or semi-autonomous vehicleinclude the one or more cameras.

In some implementations, comparing the input sensor data to the surfelrepresentation to identify the one or more differences between theobservation and the surfel representation includes performing featurematching using the image data and the surfel representation to identifyone or more features in the image data that match corresponding one ormore features in the surfel representation, the one or more features inthe surfel representation correspond to one or more groupings of surfelsin the surfel representation, the one or more differences between theobservation and the three-dimensional representation include one or moredistances between the one or more features in the image data andcorresponding one or more groupings of surfels in the surfelrepresentation, and calibrating the at least one sensor includescalibrating the one or more cameras using the one or more distances.

In some implementations, calibrating the one or more cameras includesadjusting one or more of the following: an offset of a camera of the oneor more cameras; a rotation a camera of the one or more cameras; radialdistortion parameters of a lens of a camera of the one or more cameras;or a focal length of a lens of a camera of the one or more cameras.

In some implementations, the method includes determining a type ofenvironment of the real-world environment using the surfelrepresentation, where calibrating the at least one sensor includesdetermining that calibration is viable based on the type of environment.

In some implementations, determining the type of environment of thereal-world environment using the surfel representation includes:analyzing the surfel representation or a portion of the surfelrepresentation to identify classifications of surfels in the surfelrepresentation or the portion of the surfel representation; and based onpercentages of the classifications of the surfels in the surfelrepresentation or the portion of the surfel representation, determininga type of environment from multiple types of environments.

Other embodiments of these aspects include corresponding systems,apparatus, and computer programs encoded on computer storage devices,configured to perform the actions of the methods. A system of one ormore computers can be so configured by virtue of software, firmware,hardware, or a combination of them installed on the system that, inoperation, cause the system to perform the actions. One or more computerprograms can be so configured by virtue having instructions that, whenexecuted by data processing apparatus, cause the apparatus to performthe actions.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages. Using a surfel representation to combine theexisting data and the new sensor data can be particularly efficient.Using techniques described in this specification, a system can quicklyintegrate new sensor data with the data in the surfel map to generate arepresentation that is also a surfel map. This process is especiallytime- and memory-efficient because surfels require relatively littlebookkeeping, as each surfel is an independent entity. Existing systemsthat rely, e.g., on a 3D mesh cannot integrate sensor data as seamlesslybecause if the system moves one particular vertex of the mesh, then theentire mesh is affected; different vertices might cross over each other,yielding a crinkled mesh that that must be untangled.

Moreover, numerous advantages can be realized by using a filtered surfelmap for one or more of navigation, localization, and calibration. Forexample, the disclosed system can generate a filtered surfel map thatonly includes or depicts surfels having a sufficient level ofreliability. The surfels in the filtered surfel map may be those thatrepresent objects that are unlikely to change positions and/orappearances over time. As such, navigation, localization, andcalibration accuracy can be improved due to the improved reliability ofthe surfel map that is being used to perform these functions. Forexample, the on-board system may compare collected sensor datacorresponding to a particular area in the environment to only thosesurfels that remain in the filtered surfel map to more accuratelyidentify a location for the vehicle. Accordingly, the likelihood oferrors that might otherwise be introduced due to objects in theenvironment changing position, size, and/or shape can be significantlyreduced. Similarly, errors that might otherwise be introduced to objectshaving to fine a structure can be significantly reduced.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example system.

FIG. 2A is an illustration of an example environment.

FIG. 2B is an illustration of an example surfel map of the environmentof FIG. 2A.

FIG. 3 is a flow diagram of an example process for combining surfel dataand sensor data.

FIG. 4 is a diagram illustrating an example environment.

FIG. 5A is a diagram illustrating an example environment map of at leasta portion of the environment of FIG. 4 .

FIG. 5B is a diagram illustrating an example process of filtering anenvironment map.

FIG. 5C is a diagram illustrating an example calibration procedure toupdate lidar sensor parameters using a filtered environment map.

FIG. 5D is a diagram illustrating an example calibration procedure toupdate camera parameters using a filtered environment map.

FIG. 6 is a flow diagram of an example process for using an environmentmap to calibrate one or more sensors of a system.

DETAILED DESCRIPTION

This specification describes how a vehicle, e.g., an autonomous orsemi-autonomous vehicle, can modify a surfel map that represents areal-world environment to remove or hide unreliable portions of thesurfel map. The resulting modified surfel map can be used to improve theoperation of the vehicle in the environment, by, for example, improvingthe accuracy of the perceived location of the vehicle in theenvironment, improving the accuracy of perceived locations of objects inthe environment, and improving the calibration accuracy of varioussensors of a navigation system of the vehicle. Moreover, having beensimplified with respect to the surfel map, the modified surfel map canreduce the amount of sensor data required to be collected and,relatedly, the processing requirements on the vehicle's navigationsystem.

As an example, the navigation system of the vehicle may access a surfelmap corresponding to a particular section of a road that the vehicle iscurrently traveling along. The navigation system may proceed to hide allsurfels in the surfel map that are associated with a type of object thatis deemed unreliable. For example, the navigation system may hide anysurfels that have been tagged as representing a plant, a person, ananimal, or a vehicle. The navigation system may use the resultingmodified surfel map to determine what sensor data to collect, and/or asubset of collected sensor data that is required to be analyzed. Forexample, the navigation system may only collect sensor data in adirection that corresponds to one or more objects having representationsin the modified surfel map. Similarly, the navigation system may onlyprocess sensor data corresponding to one or more objects havingrepresentations in the modified surfel map.

In this specification, a surfel is data that represents atwo-dimensional surface that corresponds to a particularthree-dimensional coordinate system in an environment. A surfel includesdata representing a position and an orientation of the two-dimensionalsurface in the three-dimensional coordinate system. The position andorientation of a surfel can be defined by a corresponding set ofcoordinates. For example, a surfel can be defined by spatialcoordinates, e.g., (x,y,z) defining a particular position in athree-dimensional coordinate system, and orientation coordinates, e.g.,(pitch, yaw, roll) defining a particular orientation of the surface atthe particular position. As another example, a surfel can be defined byspatial coordinates that define the particular position in athree-dimensional coordinate system and a normal vector, e.g., a vectorwith a magnitude of 1, that defines the orientation of the surface atthe particular position. The location of a surfel can be represented inany appropriate coordinate system. In some implementations, a system candivide the environment being modeled to include volume elements (voxels)and generate at most one surfel for each voxel in the environment thatincludes a detected object. In some other implementations, a system candivide the environment being modeled into voxels, where each voxel caninclude multiple surfels; this can allow each voxel to represent complexsurfaces more accurately.

A surfel can also optionally include size and shape parameters, althoughoften all surfels in a surfel map have the same size and shape. A surfelcan have any appropriate shape. For example, a surfel can be a square, arectangle, an ellipsoid, or a two-dimensional disc, to name just a fewexamples. In some implementations, different surfels in a surfel map canhave different sizes, so that a surfel map can have varying levels ofgranularity depending on the environment described by the surfel map;e.g., large surfels can corresponds to large, flat areas of theenvironment, while smaller surfels can represent areas of theenvironment that require higher detail.

In this specification, a surfel map is a collection of surfels that eachcorrespond to a respective location in the same environment. The surfelsin a surfel map collectively represent the surface detections of objectsin the environment. In some implementations, each surfel in a surfel mapcan have additional data associated with it, e.g., one or more labelsdescribing the surface or object characterized by the surfel. As aparticular example, if a surfel map represents a portion of a cityblock, then each surfel in the surfel map can have a semantic labelidentifying the object that is being partially characterized by thesurfel, e.g., “streetlight,” “stop sign,” “mailbox,” etc.

A surfel map can characterize a real-world environment, e.g., aparticular portion of a city block in the real world, or a simulatedenvironment, e.g., a virtual intersection that is used to simulateautonomous driving decisions to train one or more machine learningmodels. As a particular example, a surfel map characterizing areal-world environment can be generated using sensor data that has beencaptured by sensors operating in the real-world environment, e.g.,sensors on-board a vehicle navigating through the environment. In someimplementations, an environment can be partitioned into multiplethree-dimensional volumes, e.g., a three-dimensional grid of cubes ofequal size, and a surfel map characterizing the environment can have atmost one surfel corresponding to each volume.

After the surfel map has been generated, e.g., by combining sensor datagathered by multiple vehicles across multiple trips through thereal-world, one or more systems on-board a vehicle can receive thegenerated surfel map. Then, when navigating through a location in thereal world that is represented by the surfel map, the vehicle canprocess the surfel map along with real-time sensor measurements of theenvironment in order to make better driving decisions than if thevehicle were to rely on the real-time sensor measurements alone.

FIG. 1 is a diagram of an example system 100. The system 100 can includemultiple vehicles, each with a respective on-board system. Forsimplicity, a single vehicle 102 and its on-board system 110 is depictedin FIG. 1 . The system 100 also includes a server system 122 which everyvehicle in the system 100 can access.

The vehicle 102 in FIG. 1 is illustrated as an automobile, but theon-board system 110 can be located on-board any appropriate vehicletype. The vehicle 102 can be a fully autonomous vehicle that determinesand executes fully-autonomous driving decisions in order to navigatethrough an environment. The vehicle 102 can also be a semi-autonomousvehicle that uses predictions to aid a human driver. For example, thevehicle 102 can autonomously apply the brakes if a prediction indicatesthat a human driver is about to collide with an object in theenvironment, e.g., an object or another vehicle represented in a surfelmap. The on-board system 110 includes one or more sensor subsystems 120.The sensor subsystems 120 include a combination of components thatreceive reflections of electromagnetic radiation, e.g., lidar systemsthat detect reflections of laser light, radar systems that detectreflections of radio waves, and camera systems that detect reflectionsof visible light.

The sensor data generated by a given sensor generally indicates adistance, a direction, and an intensity of reflected radiation. Forexample, a sensor can transmit one or more pulses of electromagneticradiation in a particular direction and can measure the intensity of anyreflections as well as the time that the reflection was received. Adistance can be computed by determining how long it took between a pulseand its corresponding reflection. The sensor can continually sweep aparticular space in angle, azimuth, or both. Sweeping in azimuth, forexample, can allow a sensor to detect multiple objects along the sameline of sight.

The sensor subsystems 120 or other components of the vehicle 102 canalso classify groups of one or more raw sensor measurements from one ormore sensors as being measures of an object of a particular type. Agroup of sensor measurements can be represented in any of a variety ofways, depending on the kinds of sensor measurements that are beingcaptured. For example, each group of raw laser sensor measurements canbe represented as a three-dimensional point cloud, with each pointhaving an intensity and a position. In some implementations, theposition is represented as a range and elevation pair. Each group ofcamera sensor measurements can be represented as an image patch, e.g.,an RGB image patch.

Once the sensor subsystems 120 classify one or more groups of raw sensormeasurements as being measures of a respective object of a particulartype, the sensor subsystems 120 can compile the raw sensor measurementsinto a set of raw sensor data 125, and send the raw sensor data 125 toan environment prediction system 130.

The on-board system 110 also includes an on-board surfel map store 140that stores a global surfel map 145 of the real-world. The global surfelmap 145 is an existing surfel map that has been generated by combiningsensor data captured by multiple vehicles navigating through the realworld.

Generally, every vehicle in the system 100 can use the same globalsurfel map 145. In some cases, different vehicles in the system 100 canuse different global surfel maps 145, e.g., when some vehicles have notyet obtained an updated version of the global surfel map 145 from theserver system 122. The global surfel map 145 may correspond to aparticular environment. For example, the global surfel map 145 may be amap for a particular road, a section of a particular road, a particulardirection of travel on a particular road, a particular direction oftravel for a section of a particular road, etc.

Each surfel in the global surfel map 145 can have associated data thatencodes multiple classes of semantic information for the surfel. Forexample, for each of the classes of semantic information, the surfel mapcan have one or more labels characterizing a prediction for the surfelcorresponding to the class, where each label has a correspondingprobability. As a particular example, each surfel can have multiplelabels, with associated probabilities, predicting the type of the objectcharacterized by the surfel, e.g., “pole” with probability 0.8, “streetsign” with probability 0.15, and “fire hydrant” with probability 0.05.

The environment prediction system 130 can receive the global surfel map145 and combine it with the raw sensor data 125 to generate anenvironment prediction 135. The environment prediction 135 includes datathat characterizes a prediction for the current state of theenvironment, including predictions for an object or surface at one ormore locations in the environment.

In some cases, the environment prediction system 130 modifies the globalsurfel map 145 to generate a filtered surfel map 132. The environmentprediction system 130 may generate the filtered surfel map 132 by, forexample, filtering (e.g., removing or hiding) surfels in the globalsurfel map 145 that are determined to be unreliable. Surfels may beconsidered unreliable if they are associated with a particular type ofobject, e.g., a type of object that frequently changes positions,changes appearance, or has too fine a structure (e.g., leaves of atree). For example, the environment prediction system 130 may generatethe filtered surfel map 132 by hiding any surfels in the global surfelmap 145 that are associated with objects classified as plants, vehicles,equipment, animals, persons, etc. Surfels may similarly be consideredunreliable if they are associated with greater than a threshold level ofuncertainty. For example, the environment prediction system 130 maygenerate the filtered surfel map 132 by removing any surfels in theglobal surfel map 145 that have a classification confidence that isbelow 0.85.

In some cases, as described in more detail below, determining whatsurfels to filter from the global surfel map 145 to generate thefiltered surfel map 132 may be based on the location of the vehicle 102.Specifically, identifying what surfels that the environment predictionsystem 130 should remove may be based on the type of environment thatthe vehicle 102 is navigating through. For example, if the vehicle 102is traveling on a four-lane road, the environment prediction system 130may remove any surfels corresponding to plants (e.g., trees, bushes,etc.). However, if the vehicle 102 is traveling along a two-lane road ina heavily forested area, the environment prediction system 130 mayrefrain from removing or hiding any surfels corresponding to plants(e.g., trees, bushes, etc.). In this example, the environment predictionsystem 130 may further determine that no surfels should be removed orhidden from the global surfel map 145 when the vehicle 102 is travelingthrough a heavily forested area.

In some cases, as described in more detail below, determining whatsurfels to filter from the global surfel map 145 to generate thefiltered surfel map 132 may be based on the certain conditions of theenvironment that the vehicle 102 is navigating through. For example, ifthe raw sensor data 125 of the sensors subsystems 120 indicates that itis currently snowing in the environment that the vehicle 102 istraveling in, the environment prediction system 130 may remove surfelscorresponding to sidewalks to address the accumulation of snow on thesidewalk and the resulting errors that may arise from such snowaccumulation (e.g., localization errors, calibration errors, etc.).

In generating the filtered surfel map 132, the environment predictionsystem 130 can, for example, improve localization accuracy of thevehicle 102 in the environment and/or the calibration accuracy ofsensors in the sensor subsystems 120. Notably, the environmentprediction system 130 may generate the environment prediction 135 usingthe filtered surfel map 132 (e.g., in place of the global surfel map145). Therefore, the environment prediction 135 may be made based ononly a subset of surfels in the global surfel map 145 that aredetermined to be sufficiently reliable. As an example, the environmentprediction 135 may be an indication of the estimated current location ofthe vehicle 102 in the environment. This estimated location may be moreaccurate than, for example, an estimated location that was determinedusing the global surfel map 145 since distances for localization will becalculated using surfels in the filtered surfel map 132 that are morelikely to not change position (e.g., road signs, road markers, barriers,sidewalks, etc.).

Sensor calibration can similarly be improved using the filtered surfelmap 132. For example, instead of randomly comparing observed pointdistances to corresponding expected distances in the global surfel map145, the on-board system 110 can obtain point distances on specific areain the environment that correspond to surfels in the filtered surfel map132. Because these surfels are more reliable, calibration accuracy ofsensors in the sensor subsystems (e.g., lidar sensors, time-of-flightsensors, cameras, etc.) can be improved. Calibration may include theupdating of positions of the sensors, settings of the sensors, orsynchronization between two or more sensors (e.g., modifying theposition of a camera with respect to a lidar coordinate system).

In some cases, only a portion of the global surfel map 145 is modifiedto generate the filtered surfel map 132. For example, the environmentprediction system 130 may filter (e.g., remove or hide) particularsurfels in the global surfel map 145 that correspond to the environmentthat vehicle 102 is currently navigating through or is expected tonavigate through. For example, the environment prediction system 130 mayfilter out particular surfels from a portion of the global surfel map145 that corresponds to an area defined by a 100 ft behind the currentlocation of the vehicle 102, 200 ft to the right and left of the currentlocation of the vehicle 102, and 0.5 miles in front of the vehicle 102.

Similarly, the filtered surfel map 132 may be a map of only a portion ofthe global surfel map 145. For example, the filtered surfel map 132 maybe a map for an area defined by a 100 ft behind the current location ofthe vehicle 102, 200 ft to the right and left of the current location ofthe vehicle 102, and 0.5 miles in front of the vehicle 102. Theenvironment prediction system 130 may generate new filtered surfel mapsfrom the global surfel map 145 and/or the immediately preceding surfelmap as the vehicle 102 continues to navigate (e.g., the environmentprediction system 130 may start generating an updated surfel map upondetecting that the vehicle 0.25 miles from a front edge of an areacorresponding to the current filtered surfel map).

In some cases, the server system 122 generates the filtered surfel map132. For example, the server system 122 may filter different portions ofthe global surfel map 145 to generate a filtered surfel map 132. Theserver system 122 may filter different portions of the global surfel map145 differently, e.g., depending on the type of environment for each ofthe portions. The server system 122 may proceed to send the filteredsurfel map 132 to the on-board surfel map store 140.

The raw sensor data 125 might show that the environment through whichthe vehicle 102 is navigating has changed. In some cases, the changesmight be large and discontinuous, e.g., if a new building has beenconstructed or a road has been closed for construction since the lasttime the portion of the global surfel map 145 or the filtered surfel map132 corresponding to the environment has been updated. In some othercases, the changes might be small and continuous, e.g., if a bush grewby an inch or a leaning pole increased its tilt. In either case, the rawsensor data 125 can capture these changes to the world, and theenvironment prediction system 130 can use the raw sensor data to updatethe data characterizing the environment stored in the global surfel map145 or the filtered surfel map 132 to reflect these changes in theenvironment prediction 135.

For one or more objects represented in the global surfel map 145 or thefiltered surfel map 132, the environment prediction system 130 can usethe raw sensor data 125 to determine a probability that the object iscurrently in the environment. In some implementations, the environmentprediction system 130 can use a Bayesian model to generate thepredictions of which objects are currently in the environment, where thedata in the global surfel map 145 or the filtered surfel map 132 istreated as a prior distribution for the state of the environment, andthe raw sensor data 125 is an observation of the environment. Theenvironment prediction system 130 can perform a Bayesian update togenerate a posterior belief of the state of the environment, and includethis posterior belief in the environment prediction 135. In someimplementations, the raw sensor data 125 also has a probabilitydistribution for each object detected by the sensor subsystem 120describing a confidence that the object is in the environment at thecorresponding location; in some other implementations, the raw sensordata 125 includes detected objects with no corresponding probabilitydistribution.

For example, if the global surfel map 145 or the filtered surfel map 132includes a representation of a particular object, and the raw sensordata 125 includes a strong detection of the particular object in thesame location in the environment, then the environment prediction 135can include a prediction that the object is in the location with highprobability, e.g., 0.95 or 0.99. If the global surfel map 145 or thefiltered surfel map 132 does not include the particular object, but theraw sensor data 125 includes a strong detection of the particular objectin the environment, then the environment prediction 135 might include aweak prediction that the object is in the location indicated by the rawsensor data 125, e.g. predict that the object is at the location withprobability of 0.5 or 0.6. If the global surfel map 145 or the filteredsurfel map 132 does include the particular object, but the raw sensordata 125 does not include a detection of the object at the correspondinglocation, or includes only a weak detection of the object, then theenvironment prediction 135 might include a prediction that has moderateuncertainty, e.g. assigning a 0.7 or 0.8 probability that the object ispresent.

That is, the environment prediction system 130 might assign moreconfidence to the correctness of the global surfel map 145 or thefiltered surfel map 132 than to the correctness of the raw sensor data125. In some other implementations, the environment prediction system130 might assign the same or more confidence to the correctness of theraw sensor data 125 than to the correctness of the global surfel map 145or the filtered surfel map 132. In either case, the environmentprediction system 130 does not treat the raw sensor data 125 or theglobal surfel map 145 or the filtered surfel map 132 as a ground-truth,but rather associates uncertainty with both in order to combine them.Approaching each input in a probabilistic manner can generate a moreaccurate environment prediction 135, as the raw sensor data 125 mighthave errors, e.g., if the sensors in the sensor subsystems 120 aremiscalibrated, and the global surfel map 145 or the filtered surfel map132 might have errors, e.g., if the state of the world has changed.

In some implementations, the environment prediction 135 can also includea prediction for each class of semantic information for each object inthe environment. For example, the environment prediction system 130 canuse a Bayesian model to update the associated data of each surfel in theglobal surfel map 145 or the filtered surfel map 132 using the rawsensor data 125 in order to generate a prediction for each semanticclass and for each object in the environment. For each particular objectrepresented in the global surfel map 145 or the filtered surfel map 132,the environment prediction system 130 can use the existing labels ofsemantic information of the surfels corresponding to the particularobject as a prior distribution for the true labels for the particularobject. The environment prediction system 130 can then update each priorusing the raw sensor data 125 to generate posterior labels andassociated probabilities for each class of semantic information for theparticular object. In some such implementations, the raw sensor data 125also has a probability distribution of labels for each semantic classfor each object detected by the sensor subsystem 120; in some other suchimplementations, the raw sensor data 125 has a single label for eachsemantic class for each detected object.

Continuing the previous particular example, where a particular surfelcharacterizes a pole with probability 0.8, a street sign withprobability 0.15, and fire hydrant with probability 0.05, if the sensorsubsystems 120 detect a pole at the same location in the environmentwith high probability, then the Bayesian update performed by theenvironment prediction system 130 might generate new labels indicatingthat the object is a pole with probability 0.85, a street sign withprobability 0.12, and fire hydrant with probability 0.03. The new labelsand associated probabilities for the object are added to the environmentprediction 135.

Similarly, where a particular surfel characterizes a barrier withprobability 0.92, and a street sign with probability 0.08, if the sensorsubsystems 120 detect a barrier at the same location in the environmentwith high probability, then the Bayesian update performed by theenvironment prediction system 130 might generate new labels indicatingthat the object is a barrier with probability 0.95, and a street signwith probability 0.05. The new labels and associated probabilities forthe object are added to the environment prediction 135

The environment prediction system 130 can provide the environmentprediction 135 to a planning subsystem 150, which can use theenvironment prediction system 130 to make autonomous driving decisions,e.g., generating a planned trajectory for the vehicle 102 through theenvironment.

The planning subsystem 150 can make use of a barrier logic subsystem 152to determine whether a barrier is likely to prevent detected pedestriansfrom entering the road. As an example, the barrier logic subsystem candetermine that a barrier is sufficiently likely to prevent a detectedpedestrian from entering a roadway on which the vehicle 102 is travelingor from crossing a previously determined path for the vehicle 102. Theplanning subsystem 150 can thus determine that no changes should be madeto planned path for the vehicle 102, despite the presence of detectedpedestrians.

The environment prediction system 130 can also provide the raw sensordata 125 to a raw sensor data store 160 located in the server system122.

The server system 122 is typically hosted within a data center 124,which can be a distributed computing system having hundreds or thousandsof computers in one or more locations.

The server system 122 includes a raw sensor data store 160 that storesraw sensor data generated by respective vehicles navigating through thereal world. As each vehicle captures new sensor data characterizinglocations in the real world, each vehicle can provide the sensor data tothe server system 122. The server system 122 can then use the sensordata to update the global surfel map that every vehicle in the system100 uses. That is, when a particular vehicle discovers that the realworld has changed in some way, e.g., construction has started at aparticular intersection or a street sign has been taken down, thevehicle can provide sensor data to the server system 122 so that therest of the vehicles in the system 100 can be informed of the change.

The server system 122 also includes a global surfel map store 180 thatmaintains the current version of the global surfel map 185.

A surfel map updating system 170, also hosted in the server system 122,can obtain the current global surfel map 185 and a batch of raw sensordata 165 from the raw sensor data store 160 in order to generate anupdated global surfel map 175. In some implementations, the surfel mapupdating system 170 updates the global surfel map at regular timeintervals, e.g., once per hour or once per day, obtaining a batch of allof the raw sensor data 165 that has been added to the raw sensor datastore 160 since the last update. In some other implementations, thesurfel map updating system 170 updates the global surfel map whenever anew raw sensor data 125 is received by the raw sensor data store 160.

In some implementations, the surfel map updating system 170 generatesthe updated global surfel map 175 in a probabilistic way.

In some such implementations, for each measurement in the batch of rawsensor data 165, the surfel map updating system 170 can determine asurfel in the current global surfel map 185 corresponding to thelocation in the environment of the measurement, and combine themeasurement with the determined surfel. For example, the surfel mapupdating system 170 can use a Bayesian model to update the associateddata of a surfel using a new measurement, treating the associated dataof the surfel in the current global surfel map 185 as a priordistribution. The surfel map updating system 170 can then update theprior using the measurement to generate posterior distribution for thecorresponding location. This posterior distribution is then included inthe associated data of the corresponding surfel in the updated globalsurfel map 175.

If there is not currently a surfel at the location of a new measurement,then the surfel map updating system 170 can generate a new surfelaccording to the measurement.

In some such implementations, the surfel map updating system 170 canalso update each surfel in the current global surfel map 185 that didnot have a corresponding new measurement in the batch of raw sensor data165 to reflect a lower certainty that an object is at the locationcorresponding to the surfel. In some cases, e.g., if the batch of rawsensor data 165 indicates a high confidence that there is not an objectat the corresponding location, the surfel map updating system 170 canremove the surfel from the updated global surfel map 175 altogether. Insome other cases, e.g., when the current global surfel map 185 has ahigh confidence that the object characterized by the surfel is permanentand therefore that the lack of a measurement of the object in the batchof raw sensor data 165 might be an error, the surfel map updating system170 might keep the surfel in the updated global surfel map 175 butdecrease the confidence of the updated global surfel map 175 that anobject is at the corresponding location.

After generating the updated global surfel map 175, the surfel mapupdating system 170 can store it in the global surfel map store 180,replacing the stale global surfel map 185. Each vehicle in the system100 can then obtain the updated global surfel map 175 from the serversystem 122, e.g., through a wired or wireless connection, replacing thestale version with the retrieved updated global surfel map 175 in theon-board surfel map store 140. In some implementations, each vehicle inthe system 100 retrieves an updated global surfel map 175 whenever theglobal surfel map is updated and the vehicle is connected to the serversystem 122 through a wired or wireless connection. In some otherimplementations, each vehicle in the system 100 retrieves the mostrecent updated global surfel map 175 at regular time intervals, e.g.,once per day or once per hour.

FIG. 2A is an illustration of an example environment 200. Theenvironment 200 is depicted from the point of view of a sensor on-boarda vehicle navigating through the environment 200. The environment 200includes a sign 202, a bush 204, and an overpass 206. The on-boardsystem 110 described in FIG. 1 might classify the bush 204 as a barrieror a barrier with a bush label.

FIG. 2B is an illustration of an example surfel map 250 of theenvironment 200 of FIG. 2A.

Each surfel in the example surfel map 250 is represented by a disk, anddefined by three coordinates (latitude, longitude, altitude), thatidentify a position of the surfel in a common coordinate system of theenvironment 200 and by a normal vector that identifies an orientation ofthe surfel. For example, each voxel can be defined to be the disk thatextends some radius, e.g., 1, 10, 25, or 100 centimeters, around the(latitude, longitude, altitude) coordinate. In some otherimplementations, the surfels can be represented as other two-dimensionalshapes, e.g., ellipsoids or squares.

The environment 200 is partitioned into a grid of equal-sized voxels.Each voxel in the grid of the environment 200 can contain at most onesurfel, where, e.g., the (latitude, longitude, altitude) coordinate ofeach surfel defines the voxel that the surfel occupies. That is, ifthere is a surface of an object at the location in the environmentcorresponding to a voxel, then there can be a surfel characterizing thesurface in the voxel; if there is not a surface of an object at thelocation, then the voxel is empty. In some other implementations, asingle surfel map can contain surfels of various different sizes thatare not organized within a fixed spatial grid.

Each surfel in the surfel map 250 has associated data characterizingsemantic information for the surfel. For example, as discussed above,for each of multiple classes of semantic information, the surfel map canhave one or more labels characterizing a prediction for the surfelcorresponding to the class, where each label has a correspondingprobability. As a particular example, each surfel can have multiplelabels, with associated probabilities, predicting the type of the objectcharacterized by the surfel. As another particular example, each surfelcan have multiple labels, with associated probabilities, predicting thepermanence of the object characterized by the surfel; for example, a“permanent” label might have a high associated probability for surfelscharacterizing buildings, while the “permanent” label might have a highprobability for surfels characterizing vegetation. Other classes ofsemantic information can include a color, reflectivity, or opacity ofthe object characterized by the surfel.

For example, the surfel map 250 includes a sign surfel 252 thatcharacterizes a portion of the surface of the sign 202 depicted in FIG.2A. The sign surfel 252 might have labels predicted that the type of theobject characterized by the sign surfel 252 is “sign” with probability0.9 and “billboard” with probability 0.1. Because street signs arerelatively permanent objects, the “permanent” label for the sign surfel252 might be 0.95. The sign surfel 252 might have color labelspredicting the color of the sign 202 to be “green” with probability 0.8and “blue” with probability 0.2. Because the sign 202 is completelyopaque and reflects some light, an opacity label of the sign surfel 252might predict that the sign is “opaque” with probability 0.99 and areflectivity label of the sign surfel 252 might predict that the sign is“reflective” with probability 0.6.

As another example, the surfel map 250 includes a bush surfel 254 thatcharacterizes a portion of the bush 204 depicted in FIG. 2A. The bushsurfel 254 may be considered a barrier surfel when the bush isconsidered a barrier. The bush surfel 254 might have labels predictedthat the type of the object characterized by the bush surfel 254 is“barrier” or “bush” with probability 0.75 and “tree” with probability0.25. Because bushes can grow, be trimmed, and die with relativefrequency, the “permanent” label for the bush surfel 254 might be 0.2.The bush surfel 254 might have color labels predicting the color of thebush 204 to be “green” with probability 0.7 and “yellow” withprobability 0.3. Because the bush 204 is not completely opaque and doesnot reflect a lot of light, an opacity label of the bush surfel 254might predict that the sign is “opaque” with probability 0.7 and areflectivity label of the sign surfel 252 might predict that the sign is“reflective” with probability 0.4.

Note that, for any latitude and longitude in the environment 200, i.e.,for any given (latitude, longitude) position in a plane running parallelto the ground of the environment 200, the surfel map 250 can includemultiple different surfels each corresponding to a different altitude inthe environment 200, as defined by the altitude coordinate of thesurfel. This represents a distinction between some existing techniquesthat are “2.5-dimensional,” i.e., techniques that only allow a map tocontain a single point at a particular altitude for any given latitudeand longitude in a three-dimensional map of the environment. Theseexisting techniques can sometimes fail when an environment has multipleobjects at respective altitudes at the same latitude and longitude inthe environment. For example, such existing techniques would be unableto capture both the overpass 206 in the environment 200 and the roadunderneath the overpass 205. The surfel map, on the other hand, is ableto represent both the overpass 206 and the road underneath the overpass206, e.g., with an overpass surfel 256 and a road surfel 258 that havethe same latitude coordinate and longitude coordinate but a differentaltitude coordinate.

FIG. 3 is a flow diagram of an example process 300 for combining surfeldata and sensor data. For convenience, the process 300 will be describedas being performed by a system of one or more computers located in oneor more locations. For example, an environment prediction system, e.g.,the environment prediction system 130 depicted in FIG. 1 , appropriatelyprogrammed in accordance with this specification, can perform theprocess 300.

The system obtains surfel data for an environment (step 302). The surfeldata includes multiple surfels that each correspond to a respectivedifferent location in the environment. Each surfel in the surfel datacan also have associated data. The associated data can include acertainty measure that characterizes a likelihood that the surfacerepresented by the surfel is at the respective location of the surfel inthe environment. That is, the certainty measure is a measure of howconfident the system is that the surfel represents a surface that isactually in the environment at the current time point. For example, asurfel in the surfel map that represents a surface of a concrete barriermight have a relatively high certainty measure, because it is unlikelythat the concrete barrier was removed between the point in time when thesurfel map was created and a current time that the vehicle is navigatingthrough the environment. As another example, a surfel in the surfel mapthat represents a surface of a political campaign yard sign might have arelatively low certainty measure, because political campaign yard signsare usually temporary and therefore it is relatively likely that theyard sign has been removed between the time point at which the surfelmap was created and the current time point.

The associated data of each surfel can also include a respective classprediction for each of one or more classes of semantic information forthe surface represented by the surfel. In some implementations, thesurfel data is represented using a voxel grid, where each surfel in thesurfel data corresponds to a different voxel in the voxel grid.

In some cases, the system filters the surfel data to remove or hide oneor more surfels from the surfel data. For example, with respect to FIG.1 , the environment prediction system 130 may remove or hide surfelsfrom the global surfel map 145 that are associated with particular typesof objects, and/or surfels that have corresponding confidence that doesnot meet a threshold level of confidence. By filtering out thesesurfels, modified surfel data can be generated that includes data thatis more reliable. That is the modified surfel data is more likely tomatch observations made using sensors of the sensor subsystem 120 as thevehicle 102 navigates through the environment.

The system obtains sensor data for one or more locations in theenvironment (step 304). The sensor data has been captured by one or moresensors of a vehicle navigating in the environment, e.g., the sensorsubsystems 120 of the vehicle 102 depicted in FIG. 1 .

In some implementations, the surfel data has been generated from datacaptured by one or more vehicles navigating through the environment atrespective previous time points, e.g., the same vehicle that capturedthe sensor data and/or other vehicles.

The system determines one or more particular surfels corresponding torespective locations of the sensor data (step 306). For example, foreach measurement in the sensor data, the system can select a particularsurfel that corresponds to the same location as the measurement, if oneexists in the surfel data. For example, if laser data indicates that anobject is three meters away in a particular direction, the system canrefer to a surfel map to try and identify the corresponding surfel. Thatis, the system can use the surfel map to determine that a surfel that issubstantially three meters away in substantially the same direction islabelled as part of a road barrier.

The system combines the surfel data and the sensor data to generate anobject prediction for each of the one or more locations of the sensordata (step 308). The object prediction for a particular location in theenvironment can include an updated certainty measure that characterizeslikelihood that there is a surface of an object at the particularlocation.

In some implementations, the system performs a Bayesian update togenerate the object prediction from the surfel data and sensor data.That is, the system can, for each location, determine that theassociated data of the surfel corresponding to the location is a priordistribution for the object prediction, and update the associated datausing the sensor data to generate the object prediction as the posteriordistribution.

As a particular example, for each class of information in the surfeldata to be updated, including the object prediction and/or one or moreclasses of semantic information, the system can update the probabilityassociated with the class of information using Bayes' theorem:

${{P\left( {H{❘E}} \right)} = {\frac{P\left( {E{❘H}} \right)}{P(E)} \cdot {P(H)}}},$

where H is the class of information (e.g., whether the object at thelocation is vegetation) and E is the sensor data. Here, P(H) is theprior probability corresponding to the class of information in thesurfel data, and P(E|H) is probability of the sensors producing thatparticular sensor data given that the class of information is true.Thus, P(H|E) the posterior probability of the for the class ofinformation. In some implementations, the system can execute thiscomputation independently for each class of information.

For example, the surfel data might indicate a low likelihood that thereis a surface of an object at the particular location; e.g., there maynot be a surfel in the surfel data that corresponds to the particularlocation, or there may be a surfel in the surfel data that correspondsto the particular location that has a low certainty measure, indicatinga low confidence that there is a surface at the particular location. Thesensor data, on the other hand, might indicate a high likelihood thatthere is a surface of an object at the particular location, e.g., if thesensor data includes a strong detection of an object at the particularlocation.

In some such cases, the generated object prediction for the particularlocation might indicate a high likelihood that there is a temporaryobject at the particular location, e.g., debris on the road or a trashcan moved into the street. As a particular example, the objectprediction might include a high uncertainty score, indicating a highlikelihood that there is an object at the location, and a high‘temporary’ class score corresponding to a ‘temporary’ semantic label,indicating a high likelihood that the object is temporary. In some othersuch cases, the generated object prediction for the particular locationmight indicate a low likelihood that there is an object at theparticular location, because the system might assign a higher confidenceto the surfel data than to the sensor data. That is, the system mightdetermine with a high likelihood that the sensors identified an objectat the particular location in error. In some other such cases, thegenerated object prediction for the particular location might indicate ahigh likelihood that there is an object at the particular location,because the system might assign a higher confidence to the sensor datathan the surfel data. That is, the system might determine with a highlikelihood that the surfel data is stale, i.e., that the surfel datareflects a state of the environment at a previous time point but doesnot reflect the state of the environment at the current time point.

As another example, the surfel data might indicate a high likelihoodthat there is a surface of an object at the particular location; e.g.,there may be a surfel in the surfel data that corresponds to theparticular location that has a high certainty measure. The sensor data,on the other hand, might indicate a low likelihood that there is asurface of an object at the particular location, e.g., if the sensordata does not include a detection, or only includes a weak detection, ofan object at the particular location.

In some such cases, the generated object prediction for the particularlocation might indicate a high likelihood that there is an object at theparticular location, but that it is occluded from the sensors of thevehicle. As a particular example, if there it is precipitating in theenvironment at the current time point, the sensors of the vehicle mightonly measure a weak detection of an object at the limits of the range ofthe sensors. In some other such cases, the generated object predictionfor the location might indicate a high likelihood that there is areflective object at the location. When an object is reflective, asensor that measures reflected light, e.g., a LIDAR sensor, can fail tomeasure a detection of the object and instead measure a detection of adifferent object in the environment whose reflection is captured off ofthe reflective object, e.g., a sensor might observe a tree reflected offa window instead of observing the window itself. As a particularexample, the object prediction might include a high uncertainty score,indicating a high likelihood that there is an object at the location,and a high ‘reflective’ class score corresponding to a ‘reflectivitysemantic label, indicating a high likelihood that the object isreflective. In some other such cases, the generated object predictionfor the location might indicate a high likelihood that there is atransparent or semi-transparent object at the location. When an objectis transparent, a sensor can fail to measure a detection of the objectand instead measure a detection of a different object that is behind thetransparent object. As a particular example, the object prediction mightinclude a high uncertainty score, indicating a high likelihood thatthere is an object at the location, and a low ‘opaque’ class scorecorresponding to an ‘opacity’ semantic label, indicating a highlikelihood that the object is transparent.

As another example, the surfel data and the sensor data might “agree.”That is, they might both indicate a high likelihood that there is anobject at a particular location, or they might both indicate that thereis a low likelihood that there is an object at the particular location.In these examples, the object prediction for the particular location cancorrespond to the agreed-upon state of the world.

In some implementations, the system can use the class predictions forclasses of semantic information in the surfel data to generate theobject predictions. For example, the system can retrieve the labelspreviously assigned to an identified surfel that corresponds with adetected object location. The label may indicate that the object is abarrier with 0.91 confidence, and a street sign with 0.09 confidence.

In some implementations, the generated object prediction for eachlocation in the environment also includes an updated class predictionfor each of the classes of semantic information that are represented inthe surfel data. As a particular example, if a surfel is labeled as“asphalt” with a high probability, and the sensor data captures ameasurement directly above the surfel, then the system might determinethat the measurement characterizes another object with high probability.On the other hand, if the surfel is labeled as “hedge” with highprobability, and the sensor data captures a measurement directly abovethe surfel, then the system might determine that the measurementcharacterizes the same hedge, i.e., that the hedge has grown.

In some implementations, the system can obtain multiple sets of sensordata corresponding to respective iterations of the sensors of thevehicle (e.g., spins of the sensor). In some such implementations, thesystem can execute an update for each set of sensor data in a streamingfashion, i.e., executing an independent update sequentially for each setof sensor data. In some other implementations, the system can use avoting algorithm to execute a single update to the surfel data.

In some implementations, the system can use the surfel data and thesensor data to determine that the object is a barrier and is sufficientto prevent one or more objects from entering a particular road. Forexample, the on-board system 110 can use the sensor data to verify thedimensions of a barrier and/or a material of a barrier. Based on thisinformation, the on-board system 110 may determine that this barrier issufficiently likely (greater than 90%, 95%, 97%, etc.) to prevent anypedestrians from entering the roadway, but that large animals may stillpose an unacceptable risk (e.g., barrier is unlikely to prevent morethan 80%, 85%, 90%, etc. of large animals from entering the roadway).

In some implementations, the system uses the sensor data to identifyanimate objects in the environment. For example, the on-board system 110may use LIDAR and image data to identify persons and animals in theenvironment where the vehicle 102 is driving. The on-board system 110may track these objects.

In some implementations, generating an object prediction for thelocations of the sensor data includes generating a prediction using thesurfel data and the sensor data that an animate object will not enter aroadway or otherwise cross a path of travel for the vehicle. Forexample, continuing with the previous example, the on-board system 110may determine based on its previous determinations that a barrier issufficiently likely to prevent a detected pedestrian from entering theroadway that the vehicle 102 is traveling on.

After generating the object predictions, the system can process theobject predictions to generate a planned path for the vehicle (step310). For example, the system can provide the object predictions to aplanning subsystem of the system, e.g., the planning subsystem 150depicted in FIG. 1 , and the planning subsystem can generate the plannedpath. The system can generate the planned path in order to avoidobstacles that are represented in the object predictions. The planningsubsystem can also use the class predictions for one or more of theclasses of semantic information to make autonomous driving decisions,e.g., by avoiding portions of the road surface that have a likelihood ofbeing icy.

As a particular example, the vehicle may be on a first street andapproaching a second street, and a planned path of the vehicle instructsthe vehicle to make a right turn onto the second street. The surfel dataincludes surfels representing a hedge on the left side of the firststreet, such that the hedge obstructs the sensors of the vehicle frombeing able to observe oncoming traffic moving towards the vehicle on thesecond street. Using this existing surfel data, before the vehiclearrives at the second street the planning subsystem might havedetermined to take a particular position on the first street in order tobe able observe the oncoming traffic around the hedge. However, as thevehicle approaches the second street, the sensors capture sensor datathat indicates that the hedge has grown. The system can combine thesurfel data and the sensor data to generate a new object prediction forthe hedge that represents its current dimensions. The planning subsystemcan process the generated object prediction to update the planned pathso that the vehicle can take a different particular position on thefirst street in order to be able to observe the oncoming traffic aroundthe hedge.

In some cases, the system compares the surfel data and the sensor datato generate one or more other predictions. For example, with respect toFIG. 1 , the on-board system 110 can compare the raw sensor data 125 tothe filtered surfel map 132 to identify a current location for thevehicle 102 in the environment. Similarly, the on-board system 110 maycompare a subset of the raw sensor data 125 to a subset of surfels inthe filtered surfel map 132 to calibrate one or more sensors of thesensor subsystems 120.

FIG. 4 is a diagram illustrating an example environment 400. Theenvironment 400 includes a pedestrian 402, a road 404, a sidewalk 420, atree 422, a grouping of bushes 424, a flagpole 426, a flag 428 securedto the flagpole 426, and a sign 430. The road 404 can include one ormore markers, such as a road line 406 that marks an edge of the road404.

A vehicle 102 is navigating through the environment 400 using anon-board system 110. The vehicle 102 can be a semi-autonomous or a fullyautonomous vehicle that determines and executes fully-autonomous drivingdecisions in order to navigate through the environment 400. The vehicle102 can also be a semi-autonomous vehicle that uses predictions to aid ahuman driver. For example, the vehicle 102 can autonomously apply thebrakes if a prediction indicates that a human driver is about to collidewith an object in the environment 400, e.g., the pedestrian 402, anothervehicle traveling on the road 404, etc.

The on-board system 110 can include one or more sensor subsystems. Thesensor subsystems can include a combination of components that receivereflections of electromagnetic radiation, e.g., lidar systems thatdetect reflections of laser light, radar systems that detect reflectionsof radio waves, and camera systems that detect reflections of visiblelight. The vehicle 102 is illustrated as an automobile, but the on-boardsystem 110 can be located on-board any appropriate vehicle type.

The sensor data generated by a given sensor of the on-board system 110generally indicates a distance, a direction, and an intensity ofreflected radiation. For example, a sensor of the on-board system 110can transmit one or more pulses of electromagnetic radiation in aparticular direction and can measure the intensity of any reflections aswell as the time that the reflection was received. A distance can becomputed by determining how long it took between a pulse and itscorresponding reflection. The sensor can continually sweep a particularspace in angle, azimuth, or both. Sweeping in azimuth, for example, canallow a sensor to detect multiple objects along the same line of sight.

The sensor subsystems of the on-board system 110 or other components ofthe vehicle 102 can also classify groups of one or more raw sensormeasurements from one or more sensors as being measures of an object ofa particular type. A group of sensor measurements can be represented inany of a variety of ways, depending on the kinds of sensor measurementsthat are being captured. For example, each group of raw laser sensormeasurements can be represented as a three-dimensional point cloud, witheach point having an intensity and a position. In some implementations,the position is represented as a range and elevation pair. Each group ofcamera sensor measurements can be represented as an image patch, e.g.,an RGB image patch.

Once sensor subsystems of the on-board system 110 classify one or moregroups of raw sensor measurements as being measures of a respectiveobject of a particular type, the sensor subsystems of the on-boardsystem 110 can compile the raw sensor measurements into a set of rawsensor data, and send the raw data to an environment prediction system,e.g., the environment prediction system 130 shown in FIG. 1 .

The on-board system 110 can store a global surfel map, e.g., the globalsurfel map 145 shown in FIG. 1 and described above. The global surfelmap can be an existing surfel map that has been generated by combiningsensor data captured by multiple vehicles navigating through the realworld. A portion of the global surfel map can correspond to theenvironment 400, e.g., previously generated by combining sensor datacaptured by one or more vehicles had navigated through the environment400. As an example, this global surfel map can include an indication ofthe road 404, the road 404's markers including the road line 406, andthe barrier 408.

Each surfel in the global surfel map can have associated data thatencodes multiple classes of semantic information for the surfel. Forexample, for each of the classes of semantic information, the surfel mapcan have one or more labels characterizing a prediction for the surfelcorresponding to the class, where each label has a correspondingprobability. As a particular example, each surfel of the global surfelmap can have multiple labels, with associated probabilities, predictingthe type of the object characterized by the surfel, e.g., “concretebarrier” with probability 0.8, “road” with probability 0.82, and “roadline” with probability 0.91.

The environment prediction system 130 shown in FIG. 1 can receive theglobal surfel map and combine it with the raw sensor data collectedusing the on-board system 110 to generate an environment prediction forthe environment 400. The environment prediction can include data thatcharacterizes a prediction for the current state of the environment 400,including predictions for an object or surface at one or more locationsin the environment 400. The environment prediction can include alocation prediction of where the vehicle 102 currently is with respectto the environment 400.

The raw sensor data might show that the environment through which thevehicle 102 is navigating has changed. In some cases, the changes mightbe large and discontinuous, e.g., if a new building has been constructedor a road has been closed for construction since the last time theportion of the global surfel map corresponding to the environment 400has been updated. In some other cases, the changes might be small andcontinuous, e.g., if a bush grew by an inch or a leaning pole increasedits tilt. In either case, the raw sensor data can capture these changesto the world, and the environment prediction system 130 shown in FIG. 1can use the raw sensor data to update the data characterizing theenvironment 400 stored in the global surfel map to reflect these changesin the environment prediction for the environment 400.

In some implementations, certain changes in the environment 400 asindicated by the raw sensor data are not used to update the datacharacterizing the environment 400 stored in the global surfel map. Forexample, temporary objects such as pedestrians, animals, bikes,vehicles, or the like may can be identified and intentionally not beadded to the global surfel map due to their high likelihood of moving todifferent locations over time.

For one or more objects represented in the global surfel map, theenvironment prediction system 130 shown in FIG. 1 can use the raw sensordata to determine a probability that a given object is currently in theenvironment 400. In some implementations, the environment predictionsystem 130 can use a Bayesian model to generate the predictions of whichobjects are currently in the environment 400, where the data in theglobal surfel map is treated as a prior distribution for the state ofthe environment 400, and the raw sensor data is an observation of theenvironment 400. The environment prediction system 130 can perform aBayesian update to generate a posterior belief of the state of theenvironment 400, and include this posterior belief in the environmentprediction. In some implementations, the raw sensor data also has aprobability distribution for each object detected by the sensorsubsystem of the on-board system 110 describing a confidence that theobject is in the environment 400 at the corresponding location; in someother implementations, the raw sensor data includes detected objectswith no corresponding probability distribution.

For example, if the global surfel map includes a representation of aparticular object (e.g., the sign 430), and the raw sensor data includesa strong detection of the particular object in the same location in theenvironment 400, then the environment prediction can include aprediction that the object is in the location with high probability,e.g., 0.95 or 0.99. If the global surfel map does not include theparticular object (e.g., the pedestrian 402), but the raw sensor dataincludes a strong detection of the particular object in the environment400, then the environment prediction might include a prediction withmoderate uncertainty that the object is in the location indicated by theraw sensor data, e.g. predict that the object is at the location withprobability of 0.8 or 0.7. If the global surfel map does include theparticular object, but the raw sensor data does not include a detectionof the object at the corresponding location, or includes only a weakdetection of the object, then the environment prediction might include aprediction that has high uncertainty, e.g., assigning a 0.6 or 0.5probability that the object is present.

That is, the environment prediction system 130 shown in FIG. 1 mightassign the same or more confidence to the correctness of the sensor datathan to the correctness of the global surfel map. This might be true forobjects that are determined to be temporary, e.g., pedestrians, animals,vehicles, or the like. Additionally, or alternatively, the environmentprediction system 130 shown might assign more confidence to thecorrectness of the global surfel map than to the correctness of the rawsensor data. This might be true for objects that are determined to bepermanent, e.g., roads, road markers, barriers, trees, road signs,sidewalks, or the like. In either case, the environment predictionsystem 130 does not treat the raw sensor data or the global surfel mapas a ground-truth, but rather associates uncertainty with both in orderto combine them. Approaching each input in a probabilistic manner cangenerate a more accurate environment prediction, as the raw sensor datamight have errors, e.g., if the sensors in the sensor subsystems of theon-board system 110 are miscalibrated, and the global surfel map mighthave errors, e.g., if the state of the environment 400 has changed.

In some implementations, the environment prediction can also include aprediction for each class of semantic information for each object in theenvironment. For example, the environment prediction system 130 shown inFIG. 1 can use a Bayesian model to update the associated data of eachsurfel in the global surfel map using the raw sensor data in order togenerate a prediction for each semantic class and for each object in theenvironment 400. For each particular object represented in the globalsurfel map, the environment prediction system 130 can use the existinglabels of semantic information of the surfels corresponding to theparticular object as a prior distribution for the true labels for theparticular object. For example, as will be described in more detail withrespect to FIG. 5C, the on-board system 110 can assign a high confidenceto surfels in the global surfel map that are already labeled ascorresponding to the barrier 408 if the raw sensor data indicates thatthe barrier 408 is still present. The environment prediction system 130can then update each prior using the raw sensor data to generateposterior labels and associated probabilities for each class of semanticinformation for the particular object. In some such implementations, theraw sensor data also has a probability distribution of labels for eachsemantic class for each object detected by the sensor subsystem of theon-board system 110; in some other implementations, the raw sensor datahas a single label for each semantic class for each detected object.

As an example, where a particular surfel of the global surfel mapcharacterizes the sign 430 with probability 0.8 and the sidewalk 420with probability 0.2, if the sensor subsystems of the on-board system110 detect the sign 430 at the same location in the environment 400 withhigh probability, then the Bayesian update performed by the environmentprediction system 130 shown in FIG. 1 might generate new labelsindicating that the object is the sign 430 with probability 0.85 and thesidewalk 420 with probability 0.15. The new labels and associatedprobabilities for the object are added to the environment prediction.

With respect to FIG. 1 , the environment prediction system 130 canprovide the environment prediction to the planning subsystem 150, whichcan use the environment prediction to make autonomous driving decisionsfor the vehicle 102, e.g., generating a planned trajectory for thevehicle 102 through the environment 400.

FIG. 5A is a diagram illustrating an example environment map of at leasta portion of the environment of FIG. 4 . Specifically, FIG. 5Aillustrates the global surfel map 145 of the environment 400. As shown,the global surfel map 145 can include a multitude of surfels therepresent various objects in the environment 400. The objects that thesurfels represent may be those objects that are determined to have asufficient level of permanence. For example, the global surfel map 145may not include any surfels that represent animate objects such aspersons and animals.

An environment map may be a representation of a real-world environment.The environment map may be a three-dimensional representation of areal-world environment. The environment map may be a surfel map such asthe global surfel map 145. In some cases, the environment map is not asurfel map. For example, the environment map may be a polygon model.

With respect to FIG. 4 , the global surfel map 145 may be stored on theon-board system 110 or may be accessed by the on-board system 110. Theglobal surfel map 145 may have been generated prior to the vehicle 102entering the environment 400. For example, the global surfel map 145 canbe generated offline (e.g., by a server system) using collected sensordata, such as sensor data collected by one or more autonomous vehicles.

Each surfel in the global surfel map 145 is represented by a disk, anddefined by three coordinates (latitude, longitude, and altitude), thatidentify a position of the surfel in a common coordinate system of theenvironment 400 and by a normal vector that identifies an orientation ofthe surfel. For example, each volume element (voxel) can be defined tobe the disk that extends some radius, e.g., 1, 10, 25, or 100centimeters, around the coordinate (latitude, longitude, and altitude).In some other implementations, the surfels can be represented as othertwo-dimensional shapes, e.g., ellipsoids, squares, rectangles, etc.

Each surfel in the global surfel map 145 has associated datacharacterizing semantic information for the surfel. For example, asdiscussed above with respect to FIG. 1 , for each of multiple classes ofsemantic information, the global surfel map 145 can have one or morelabels characterizing a prediction for the surfel corresponding to theclass, where each label has a corresponding probability. As a particularexample, each surfel can have multiple labels, with associatedprobabilities, predicting the type of the object characterized by thesurfel. As another particular example, each surfel can have multiplelabels, with associated probabilities, predicting the permanence of theobject characterized by the surfel; for example, a “permanent” labelmight have a high associated probability for surfels characterizingbuildings, while the “temporary” label might have a low probability forsurfels characterizing vegetation. Other classes of semantic informationcan include a color, reflectivity, or opacity of the objectcharacterized by the surfel.

As illustrated, a surfel classification key 502 a indicates differentsurfel classifications and corresponding color/fill patterns forrepresentations of surfels (e.g., in the global surfel map 145). A firstclassification of surfels corresponds to representations of signs orother metal object. Surfels that are associated with the firstclassification are depicted in the global surfel map 145 having thedarkest color/the densest fill pattern. For example, surfelscorresponding to the sign 430 and the flagpole 426 in the environment400 are associated with the first classification. That is, the surfelscorresponding to the sign 430 and the flagpole 426 have been classified(e.g., by the surfel map updating system 170 shown in FIG. 1 ) asrepresenting a sign object and/or a metal object.

A second classification of surfels corresponds to representations ofroadways. Surfels that are associated with the second classification aredepicted in the global surfel map 145 having the second darkestcolor/the second densest fill pattern. For example, surfelscorresponding to the road 404 in the environment 400 are associated withthe second classification. That is, the surfels corresponding to theroad 404 have been classified (e.g., by the surfel map updating system170 shown in FIG. 1 ) as representing a roadway. The secondclassification may be more specific in that it might be reserved forsurfels representing a roadway that the vehicle 102 is currentlytraveling on.

A third classification of surfels corresponds to representations ofvegetation (e.g., grass, bushes, trees, etc.). Surfels that areassociated with the third classification are depicted in the globalsurfel map 145 having the third darkest color/the third densest fillpattern. For example, surfels corresponding to the tree 422 and thebushes 424 in the environment 400 are associated with the thirdclassification. That is, the surfels corresponding to the tree 422 andthe bushes 424 have been classified (e.g., by the surfel map updatingsystem 170 shown in FIG. 1 ) as representing some type of vegetation.The third classification can be, in some cases, broken down intomultiple classifications or sub-classifications that correspond todifferent types of vegetation. For example, there can be a firstsubclass for surfels representing grass, a second subclass for surfelsrepresenting a bush, and a third subclass for surfels representing atree.

A fourth classification of surfels corresponds to representations oftextiles (e.g., cloth, clothing, flags, etc.). Surfels that areassociated with the fourth classification are depicted in the globalsurfel map 145 having the fourth darkest color/the fourth densest fillpattern. For example, surfels corresponding to the flag 428 in theenvironment 400 are associated with the fourth classification. That is,the surfels corresponding to the flag 428 have been classified (e.g., bythe surfel map updating system 170 shown in FIG. 1 ) as representingsome type of textile. The fourth classification can be, in some cases,broken down into multiple classifications or sub-classifications thatcorrespond to different types of textiles.

A fifth classification of surfels corresponds to representations ofsidewalks and/or objects made from concrete (e.g., road barriers,walkways, etc.). Surfels that are associated with the fifthclassification are depicted in the global surfel map 145 having thelightest color/the least dense fill pattern. For example, surfelscorresponding to the sidewalk 420 in the environment 400 are associatedwith the fifth classification. That is, the surfels corresponding to thesidewalk 420 have been classified (e.g., by the surfel map updatingsystem 170 shown in FIG. 1 ) as representing a sidewalk object and/or aconcrete object. The fifth classification can be, in some cases, brokendown into multiple classifications or sub-classifications thatcorrespond to different types of concrete objects. For example, theremay be a first subclass for sidewalks (and other walkways) and a secondsubclass for road barriers.

The classifications depicted in the example surfel classification key502 a are not necessarily comprehensive. There may more or lessclassifications that surfels in the global surfel map 145 can beassociated with. As an example, there may be classification for roadbarriers, a classification for road lines, different classifications fordifferent types of vegetation (e.g., bush classification, treeclassification, grass classification, etc.), etc. In some cases, asurfel may be associated with multiple classifications in the surfelclassification key 502 a. For example, a grouping of surfels thatcorrespond to the road line 406 may be classified as both a roadway anda road line.

In some cases, the classifications depicted in the example surfelclassification key 502 a are a particular type of classification whereeach surfel in the global surfel map 145 is associated with only onethese classifications. This type of classification may be usedparticularly for generating a filtered surfel map, and/or for selectingobjects to be used during sensor calibration.

In some cases, the depiction of each surfel in the global surfel map 145corresponds to the classification with the highest level confidence foreach surfel. For example, a first surfel of the global surfel map 145may be classified as belonging to the fifth classification (e.g.,sidewalk/concrete classification) with a confidence of 0.72, and to thethird classification (e.g., grass/bush/tree classification) with aconfidence of 0.28. Accordingly, the depiction of the surfel maycorrespond to the fifth classification due to it being associated withthe higher confidence. Similarly, with respect to FIG. 1 , theenvironment prediction system 130, in generating the filtered surfel map132, may simplify global surfel map 145 such that each surfel isassociated with a single classification (e.g., the classification withthe highest confidence; or the classification of the classifications inthe surfel classification key 502 a with the highest confidence suchthat other classifications not in the surfel classification key 502 aare ignored and/or otherwise left alone). This may be helpful to reducethe processing burden of the environment prediction system 130 whengenerating the filtered surfel map 132, which could help to decrease thetime it takes for the environment prediction system 130 to generate thefiltered surfel map 132 and/or to free up processing power that can beused for other tasks such as object detection during autonomousnavigation.

The definitions of semantic labels, such as the labels “permanent”and/or “temporary” or the classifications in the surfel classificationkey 502 a, can each have one or more definitions. The definition appliedmay be dependent on context. These definitions may be set by, forexample, a system administrator. As a particular example, the label“permanent” may not necessarily have a single standard of longevity. Insome cases, an additional or alternative label may be attached toobjects that are critical to navigation and/or are reliable (e.g., havepositions that are unlikely to change in the immediate future) but thatare known to be moved at some point in the future. For example, thelabel “semi-permanent” may be attached to a representation of the sign430 in place of “permanent” to indicate that the sign 430 will likely bemoved at some point in the future.

FIG. 5B is a diagram illustrating an example process of filtering theglobal surfel map 145. In filtering the global surfel map 145, theenvironment prediction system 130 of the on-board system 110 shown inFIG. 1 can remove one or more surfels from the global surfel map 145, ortemporarily hide one or more surfels from the global surfel map 145. Theone or more surfels that are hidden or removed may be surfels associatedwith a particular classification. Additionally, or alternatively, theone or more surfels that are hidden or removed may be surfels associatedwith a confidence below a threshold level of confidence (e.g., surfelswith more than an acceptable level of uncertainty are removed or hiddenfrom the global surfel map 145).

In filtering the global surfel map 145, the environment predictionsystem 130 may provide the global surfel map 145 to a filter module 530.The filter module 530 may determine a type of environment that thevehicle 102 is located. That is, the filter module 530 may identify atype of environment corresponding to the environment 400. The type ofenvironment indicate, for example, one or more of a type of roadway thatthe vehicle 102 is traveling on (e.g., one-way street, two-lane road,four-lane highway, bridge, etc.) and/or the surrounding area (e.g.,forested, urban, suburban, etc.). As an example, the filter module 530may determine that the environment 400 is a four-lane road in a suburbanarea. In some cases, the type of environment may additionally oralternatively indicate other information such as current conditions ofthe environment (e.g., weather conditions, construction zones, closedlanes, etc.).

The filter module 530 may make this determination by analyzing theglobal surfel map 145. For example, if the percent of surfels associatedwith the second classification (e.g., roadway) is greater than 60% ofthe total surfels representing the environment and the percent ofsurfels associated with the third classification (e.g., grass/bush/tree)is between 5% and 30% of the total surfels representing the environment,then the filter module 530 may determine that the environment is ahighway. Similarly, if the percent of surfels associated with the thirdclassification (e.g., grass/bush/tree) is greater than 55%, then thefilter module 530 may determine that the environment is a forested area.As another example, if the surfels associated with the secondclassification (e.g., roadway) indicates a road width is between 18.5feet and 28 feet, the filter module 530 may determine that theenvironment is a two-lane road.

In some cases, the filter module 530 can determine multiple types ofenvironments for a particular environment. For example, the filtermodule 530 may analyze a global surfel map to determine that the vehicle102 is currently traveling down a two-lane road based on the globalsurfel map indicating that the width of the road is twenty-five feet,and that the vehicle 102 is traveling in a forested area based on 62% ofthe surfels in the environment being associated with the thirdclassification (e.g., grass/bush/tree).

An administrator may define the different types of environments.Additionally, or alternatively, an administrator may label differentgeographic areas (e.g., in the global surfel map 145 or another map thatcan be compared to the global surfel map 145) as being of a particulartype of environment. The filter module 530 may compare the location ofthe vehicle 102 to these geographic areas to identify the geographicarea that the vehicle 102 is located in (or is about to enter), andlookup a corresponding type of environment. For example, the filtermodule 530 may obtain data from a GPS unit of the on-board system 110and use the data to determine a location for vehicle 102. The filtermodule 530 may use the location for the vehicle 102 to determine thatthe vehicle 410 is in the environment 400. The filter module 530 mayproceed to lookup one or more environment type labels associated withthe environment 400. For example, the filter module 530 may access alabel indicating that the environment 400 is four-lane road.

Alternatively, the on-board system 110 or the server system 122 shown inFIG. 1 may use one or more machine learning networks to define thedifferent types of environments (e.g., to generate surfel classificationpercent ranges to associate with different types of environments),

In some cases, one or more machine learning networks are used todetermine the type of environment. For example, the filter module 530may provide the global surfel map 145 or a portion of the global surfelmap 145 that corresponds to the environment 400 as input to one or moremachine learning networks. The one or more machine learning networks mayoutput at least one value that corresponds to one or more types ofenvironments. For example, the one or more machine learning networks mayoutput a value of 0.65 The one or more machine learning networks may betrained with surfel maps or portions of surfel maps that have alreadybeen associated with one or more types of environments (e.g., where anadministrator has indicated the type of environment(s) in the surfel mapor the portions of the surfel map).

In some cases, determining the type of environment is based on otherinformation such as sensor data collected by the vehicle 102. Forexample, the current speed of the vehicle, the average speed of thevehicle 102 over an amount of time (e.g., one minute, two minutes, fiveminutes, etc.) or a particular distance (e.g., last 0.5 miles, 1.0miles, 2 miles, etc. of travel distance), or the highest speed of thevehicle over an amount of time (e.g., one minute, two minutes, fiveminutes, etc.) or a particular distance (e.g., last 0.5 miles, 1.0miles, 2 miles, etc. of travel distance) may be used by the filtermodule 530 to determine the type of environment. Specifically, if theaverage speed of the vehicle 102 over the last two minutes if 63 mph,then the filter module 530 may use the average speed in determining theenvironment that vehicle 102 is traveling in is a highway.

In some cases, the type of environment may indicate particularconditions in the environment 400. For example, the type of environmentmay indicate weather conditions that vehicle 102 is experiencing in theenvironment 400, such as rain, snow, sleet, hail, etc.

After determining the type of environment corresponding to theenvironment 400, the filter module 530 may determine one or more surfelclassifications that are unreliable based on the type of environment. Asan example, based on the environment 400 being a four-lane roadway in asuburban area, the filter module 530 may determine that surfelscorresponding to the third classification (e.g., grass/bush/tree) andthe fourth classification (e.g., flag/cloth/clothing) should be removedor hidden from the global surfel map 145. In making this determination,the environment 400 may refer to a lookup table that indicates thatclassifications of surfels that should be removed for different types ofenvironments and/or combinations of different types of environments. Asan example, a lookup table may indicate that surfels corresponding tothird classification should not be filtered when the environment is aheavily forested area (e.g., to account for the lack of non-vegetationlandmarks in heavily forested areas).

In some cases, the filter module 530 provides the type(s) ofenvironment(s) as input to one or more machine learning networks. Theoutput of the one or more machine learning networks may include a valuethat is indicative of one or more surfel classifications that should befiltered. The one or more machine learning networks may be initiallytrained using, for example, a pre-defined lookup table. The one or moremachine learning networks may be trained or retrained using one or morefeedback loops with the goal of reducing navigation errors, localizationerrors, and/or calibration errors (e.g., that arise after using thefiltered surfel map 132).

Some surfel classifications may be generally unreliable due to thembeing associated with objects that are not permanent, tend to changeposition over time, and/or tend to change appearance over time. Forexample, surfels belonging to the third classification (e.g.,grass/bush/tree) may generally be considered unreliable for tending tochange position and/or appearance over time due to growth of thevegetation, trimming of the vegetation, death of the vegetation,relocation of the vegetation, etc. Similarly, surfels belonging to thefourth classification (e.g., flag/cloth/clothing) may generally beconsidered unreliable for tending to change appearance and/or positiondue to, for example, wind or other weather conditions. However, in somecases, the filter module 530 may refrain from filtering surfelscorresponding to these classifications if it would mean, for example,filtering out too much of global surfel map 145 (e.g., more than 30%,45%, 60%, etc. of the global surfel map 145's surfels), would likelyresult in filtering out too much of the global surfel map 145 (e.g., asindicative by the type of environment), would likely result in aninsufficient number of landmarks to use for navigation, localization,and/or calibration (e.g., as indicative by the type of environment),etc.

After determining the surfel classifications that should be filteredfrom the global surfel map 145, the filter module 530 may proceed togenerate the filtered surfel map 132 by removing or hiding surfels thatare considered unreliable. That is, the filter module 530 may proceed toremove or hid surfels in the global surfel map 145 that are associatedwith the determined unreliable surfel classifications. As shown, thefiltered surfel map 132 does not include or does not depict surfelsbelonging to the third classification (e.g., grass/bush/tree) andsurfels belonging to the fourth classification (e.g.,flag/cloth/clothing). Notably, a grouping of surfels that representedthe flag 428 has been removed or hidden due to being associated with thefourth classification. The surfel classification key 502 b has also beenupdated to reflect the particular surfel classifications that have beenfiltered.

In some cases, the filter module 530 may refrain from removing or hidingany surfels based on the determined type of environment. For example, ifthe type of environment is a heavily forested area (e.g., without muchof any landmarks other than trees or other vegetation), then the filtermodule 530 may determine that no surfels should be filtered from theglobal surfel map 145. Accordingly, the filter module 530 may output anindication to the environment prediction system 130 that the on-boardsystem 110 should use the existing global surfel map 145 in theenvironment 400, e.g., for navigation, localization, and/or calibration.

In some cases, as described above, instead of filtering surfels based onthe type of environment, the filter module 530 may filter (e.g., removeor hide) surfels associated with an uncertainty (e.g., classificationuncertainty/confidence) that is below a threshold uncertainty level.

In some cases, instead of filtering surfels, the filter module 530identifies reliable and/or unreliable surfels. For example, the filtermodule 530 may, using the techniques described above, identify a subsetof surfels in the global surfel map 145 that are sufficiently reliableto use for calibration. Specifically, the filter module 530 may identifyall surfels associated with the first surfel classification (e.g.,sign/metal) as being sufficiently reliable to use to calibrate one ormore on-board sensors of the vehicle 102's on-board system 110. Thefilter module 530 may optionally tag the surfels in the subset ofsurfels to indicate that they are sufficiently reliable to use forcalibration. A calibration module 532, described below with respect toFIGS. 5C-5D, may identify the subset of surfels using the tags andproceed to use only those surfels for sensor calibration.

Reliability may also be based on the observed consistency in thelocation of particular objects (e.g., in addition to the reliabilityassociated with particular types of objects) in a real-worldenvironment. For example, the number of times that autonomous orsemi-autonomous vehicles have confirmed that the location of the sign430 in the environment 400, and/or how recently the autonomous orsemi-autonomous vehicles have confirmed that the location of the sign430 in the environment 400 may be used by the on-board system 110 indetermining a reliability for the grouping of surfels in the globalsurfel map 145 that represent the sign 430 and/or may serve as a factorin determining a reliability to associate with surfels that belong tothe first surfel classification (e.g., sign/metal).

FIG. 5C is a diagram illustrating an example calibration procedure toupdate lidar sensor parameters using a filtered environment map.Specifically, FIG. 5C is a diagram illustrating an example calibrationprocedure to update lidar sensor parameters using the filtered surfelmap 132. The lidar sensor may be part of the on-board system 110.Specifically, the lidar sensor may be part of the sensor subsystems 120shown in FIG. 1 . The calibration procedure may include performing lidardense calibration by matching points from lidar data to correspondingsurfels in in the filtered surfel map 132.

As illustrated a calibration module 532 of the environment predictionsystem 130 may receive lidar sensor parameters 534 a, raw lidar sensordata 524, the filtered surfel map 132, and location sensor data 536 asinput. In some cases, the calibration module 532 also receives imagedata corresponding to the raw lidar sensor data 524 (e.g., images thatare synchronized to the raw lidar sensor data 524) as input. The lidarsensor parameters 534 a may include, for example, a range offsetcorresponding to the range of laser points, an x-axis rotation offsetcorresponding to a roll angle offset (e.g., with respect to a positionof one or more cameras of the on-board system 110), a y-axis rotationoffset corresponding to a pitch angle offset (e.g., with respect to aposition of one or more cameras of the on-board system 110), a z-axisrotation offset corresponding to a yaw angle offset (e.g., with respectto a position of one or more cameras of the on-board system 110), anx-axis offset corresponding to a position offset along a first axis(e.g., with respect to a position of one or more cameras of the on-boardsystem 110), a y-axis offset corresponding to a position offset along asecond axis (e.g., with respect to a position of one or more cameras ofthe on-board system 110), and a z-axis offset corresponding to aposition offset along a third axis (e.g., with respect to a position ofone or more cameras of the on-board system 110). The lidar sensorparameters 534 a may include one or more additional parameters. Forexample, as described in more detail below, the lidar sensor parameters534 a may include characteristics and/or current settings of the lidarsensor.

The raw lidar sensor data 524 may include laser data such as point datathat can be used by the calibration module 532 to generate a point cloudfor the environment 400 or a portion of the environment 400. The rawlidar sensor data 524 may be generated by the lidar sensor. In somecases, the environment prediction system 130 sends instructions to thelidar sensor (e.g., to the sensor subsystems 120) to collect raw lidarsensor data 524 after the filtered surfel map 132 is generated (e.g., inresponse to generating the filtered surfel map 132).

The location sensor data 536 may be obtained using one or more sensorsof the on-board system 110. For example, the location sensor data 536may include GPS coordinates obtained using a GPS unit of the sensorsubsystems 120. Additionally, or alternatively, the location sensor data536 may include accelerometer data and/or inertial measurement unit(IMU) data that can be used estimate a location of the vehicle 102 inthe environment 400. The location sensor data 536 may be synchronizedwith the raw lidar sensor data 524 such that, for example, the locationsensor data 536 indicates the location(s) of the vehicle 102 in theenvironment 400 at a time when the raw lidar sensor data 524 wascollected, or times when different portions of the raw lidar sensor data524 was collected. The location sensor data 536 may be modified toaccount for differences in position of the lidar sensor with respect toa GPS unit used to generate the location sensor data 536.

The calibration module 532 may use sensor data corresponding to an areaof an environment to identify a corresponding section of the filteredsurfel map 132. For example, the calibration module 532 can use the rawlidar sensor data 524 (or a portion of the raw lidar sensor data 524)and/or the location sensor data 536 to identify a corresponding portionof the filtered surfel map 132. Specifically, the calibration module 532may select a subset of points in the raw lidar sensor data 524, identifyangles and/or ranges corresponding to those points (e.g., a position ofthe lidar sensor and/or a position of the vehicle 102 when the pointswere obtained), and use the location of the vehicle 102 and theidentified angles and/or ranges to identify an section 525 a in thefiltered surfel map 132 corresponding to the subset of points. Thesubset of points may be those that were obtained by the lidar sensor ata particular point in time, and/or while the lidar sensor was in aparticular position.

Additionally, or alternatively, the calibration module 532 can use theraw lidar sensor data 524 (or a portion of the raw lidar sensor data524) with corresponding image data to identify the section 525 a.Specifically, the calibration module 532 may identify an image and asubset of points corresponding to the image, e.g., points that fallwithin the bounds of the image (e.g., where the bounds of the imagecorresponds to the section 525 a) and/or that were obtained at the sametime as when the image was taken. The calibration module 532 may proceedto determine a position of a camera synchronized to the lidar sensor andused to take the image. The calibration module 532 can use the cameraposition with the location of the vehicle 102 to identify the section525 a corresponding to the subset of points corresponding to the image.

After identifying the section 525 a, the calibration module 532 maycompare sensor data to data retrieved from the filtered surfel map 132to determine differences between the area and the section. Comparing thesensor data to data retrieved from the filtered surfel map 132 mayinclude identifying point matches between the raw lidar sensor data 524and the filtered surfel map 132. For example, the calibration module 532may match the point cloud data from the raw lidar sensor data 524 todistances of surfels in the filtered surfel map 132 to identify asection of the filtered surfel map 132 that matches the ranges in thepoint cloud data. The calibration module 532 may calculated distances byidentifying a surfel in the filtered surfel map 132 corresponding to agiven point (e.g., a surfel that represents an area in the environment400 that the corresponding laser point would contact), accessing alocation corresponding to the surfel (e.g., three-dimensionalcoordinates), and calculating a distance from surfel location to thevehicle location or location of the lidar sensor (e.g., locationindicated by the location sensor data 536).

After matching or substantially matching a sufficient number of pointsin the point cloud (e.g., a threshold number of points, a thresholdpercent of points, a threshold number or percent of points from a subsetof points sampled from the point cloud, all of the points in the pointcloud, or all of the points in a subset of points sampled from the pointcloud), the calibration module 532 may identify a section 525 b thatrepresents a section of the filtered surfel map 132 that the raw lidarsensor data 524 matches or substantially matches. That is, the section525 b may represent an area of the environment 400 that accuratelycorresponds to the raw lidar sensor data 524 or, at least, moreaccurately corresponds to the raw lidar sensor data 524 than an area ofthe environment 400 corresponding to the section 525 a.

The calibration module 532 may compare the section 525 a to the section525 b to identify differences between the section 525 a and the section525 b. These differences may include differences in location, size,shape, etc. These differences may represent the differences between theexpected locations point data with respect to the filtered surfel map132 and the actual locations of the point data with respect to thefiltered surfel map 132. The calibration module 532 may use thesedifferences to generate the updated lidar sensor parameters 534 b.

Comparing the sensor data to data retrieved from the filtered surfel map132 may alternatively or additionally include comparing the sensor datato data retrieved from the section of the filtered surfel map 132 toidentify differences between the sensor data and the data retrieved. Forexample, the calibration module 532 may compare ranges of points in apoint cloud corresponding to the section 525 a (e.g., from the raw lidarsensor data 524) to corresponding expected ranges based on the locationsof surfels in the filtered surfel map 132. In some cases, thecalibration module 532 may randomly sample points in the point cloudcorresponding to the section 525 a to obtain a subset of points and usethis subset of points for the comparison. That is, the calibrationmodule 532 may identify a range for each point in the subset of points,identify an expected range for each point in the subset of points usinglocations of surfels in the filtered surfel map 132, and proceed tocompare the ranges to the expected ranges. Identifying an expected rangefor a point in the point cloud corresponding to the section 525 a mayinclude identifying a surfel in the filtered surfel map 132corresponding to the point (e.g., the surfel represents an area in theenvironment 400 that the laser point would contact), accessing alocation corresponding to the surfel, and calculating a distance fromsurfel location to the vehicle location (e.g., location indicated by thelocation sensor data 536).

The calibration module 532 may proceed to use these differences inaddition to or in place of the points matches described above togenerate the updated lidar sensor parameters 534 b. For example, thecalibration module 532 may increase the range offset from 0.0 m to 0.08m, the x-axis rotation offset to 2°, the y-axis rotation offset to 8°,the z-axis rotation offset to 1°, the x-axis offset to 0.11 m, they-axis offset to 0.04 m, and the z-axis offset to 0.02 m. The effectthat the updated lidar sensor parameters 534 b have can be depicted bythe corrected section 525 b, that more accurately represents the area inthe environment 400 that the raw lidar sensor data 524 corresponds to.

In some cases, updating the lidar sensor parameters includes adjustingone or more lidar sensor settings or characteristics. For example, theoutput of the calibration module 532 may indicate that the one or moreof the point density (e.g., if adjustable), the field of view (e.g., ifadjustable), and/or the laser repetition rate (e.g., if adjustable) ofthe lidar sensor should be modified to improve accuracy of the lidarsensor.

FIG. 5D is a diagram illustrating an example calibration procedure toupdate camera parameters using a filtered environment map. Specifically,FIG. 5D is a diagram illustrating an example calibration procedure toupdate camera parameters using the filtered surfel map 132. The processof updating camera parameters may be used in addition to the process forupdating the lidar sensor parameters described above in FIG. 5C, or inplace of the process for updating the lidar sensor parameters. Thecalibration procedure may include performing camera feature matchingbetween collected images and flagged features in the filtered surfel map132.

As illustrated the calibration module 532 of the environment predictionsystem 130 may receive camera parameters 538 a, raw image data 526, thefiltered surfel map 132, and the location sensor data 536 as input. Thecamera parameters 538 a may include, for example, an x-axis rotationoffset corresponding to a roll angle offset (e.g., with respect to amount of the camera; with respect to a position of a lidar sensor of theon-board system 110; etc.), a y-axis rotation offset corresponding to apitch angle offset (e.g., with respect to a mount of the camera; withrespect to a position of a lidar sensor of the on-board system 110;etc.), a z-axis rotation offset corresponding to a yaw angle offset(e.g., with respect to a mount of the camera; with respect to a positionof a lidar sensor of the on-board system 110; etc.), an x-axis offsetcorresponding to a position offset along a first axis (e.g., withrespect to a mount of the camera; with respect to a position of a lidarsensor of the on-board system 110; etc.), a y-axis offset correspondingto a position offset along a second axis (e.g., with respect to a mountof the camera; with respect to a position of a lidar sensor of theon-board system 110; etc.), and a z-axis offset corresponding to aposition offset along a third axis (e.g., with respect to a mount of thecamera; with respect to a position of a lidar sensor of the on-boardsystem 110; etc.). The camera parameters 538 a may include one or moreadditional parameters. For example, as described in more detail below,the camera parameters 538 a may include characteristics and/or currentsettings of the camera.

The location sensor data 536 may be obtained using one or more sensorsof the on-board system 110. For example, the location sensor data 536may include GPS coordinates obtained using a GPS unit of the sensorsubsystems 120. Additionally, or alternatively, the location sensor data536 may include accelerometer data and/or inertial measurement unit(IMU) data that can be used estimate a location of the vehicle 102 inthe environment 400. The location sensor data 536 may be synchronizedwith the raw image data 526 such that, for example, the location sensordata 536 indicates the location(s) of the vehicle 102 in the environment400 at a time when the raw image data 526 was collected, or times whendifferent images in the raw image data 526 were collected. The locationsensor data 536 may be modified to account for differences in positionof the camera with respect to a GPS unit used to generate the locationsensor data 536.

The raw image data 526 may include visible-light images, infrared-lightimages, time-of-flight camera images, or combinations different types ofimages. The raw image data 526 may captured by the camera. In somecases, the environment prediction system 130 sends instructions to thecamera (e.g., to the sensor subsystems 120) to collect the raw imagedata 526 after the filtered surfel map 132 is generated (e.g., inresponse to generating the filtered surfel map 132). The instructionsmay include instructions for the camera to be moved to one or morepositions to captures areas of the environment 400 that correspond toflagged features in the filtered surfel map 132. These flagged featuresmay include specific surfels that have been flagged, specific groupingsof surfels that have been flagged (e.g., for representing a specificobject, or particular part(s) of an object such as the corners of anobject that are easily identifiable), surfels that are associated with aparticular surfel classification, etc. Features may be flagged by, forexample, an administrator. Additionally, or alternatively, features maybe flagged using the output of one or more algorithms (e.g., one or moremachine learning algorithms) that can indicate objects and/or featuresof objects that can easily be identified (e.g., due to large size, sharpangles, permanence/unlikely to move, large features, high contrast,etc.).

As an example, one or more classifier machine learning networks may betrained to recognize interest points on an object. For example, given aparticular image (e.g., a keypoint image) the environment predictionsystem 130 may determine all or a large number of possible views of anobject in the image. The one or more classifier machine learningnetworks may include a tree classifier that is trained to recognizekeypoints in the image. For example, the tree classifier may assign oneclass per keypoint. Alternatively, the tree classifier may use a binaryapproach where it assigns each keypoint a one for yes (e.g., indicatinga match) or a zero for no (e.g., indicating not a match).

As demonstrated in the updated surfel classification key 502 c, thefirst surfel classification (e.g., sign/metal) may have been flagged forfeature matching. This indicates that any surfels in the filtered surfelmap 132 that are associated with the first surfel classification havebeen flagged for camera feature matching. In response to determiningthat this class of surfels have been flagged for camera featurematching, the environment prediction system 130 may generate and sendinstruction to the sensor subsystems 120 shown in FIG. 1 to position thecamera towards the flagged features (e.g., using the expected locationsfor the flagged features based on the filtered surfel map 132) andcapture corresponding images. For example, the camera may capture imagesthat correspond to area of the environment 400 that are represented bythe sections 540 a-540 e of the filtered surfel map 132. The calibrationmodule 532 may provide the captured images of the raw image data 526 toone or more classifier machine learning networks to identify thekeypoints in the images (e.g., the points corresponding to the flaggedfeatures). Here, the keypoints may include, for example, anoutline/outer edge of the sign 430, and an outline/outer edge of theflagpole 426.

In some cases, the calibration module 532 converts the raw image data526 to one or more vectors prior to comparison with the filtered surfelmap 132. For example, the calibration module 532 may generate a vectorfor image patches in the raw image data 526 by taking an image andsmoothing the image patch. The calibration module 532 may proceed totake pairs of points and for each pair determine if a first point in thepair is brighter than the other point. If yes, the calibration module532 may assign a “1” to the first point. If no, the calibration module532 may assign a “0” to the first point. The calibration module 532 mayoptionally bias the pairs towards the center. The calibration module 532may do this repeatedly until the image patch is boiled down to a vector.

The calibration module 532 may use sensor data corresponding to an areaof an environment to identify a corresponding section of the filteredsurfel map 132. For example, as discussed above, the sections 540 a-540e are sections of the filtered surfel map 132 that were expected tomatch or substantially match the areas of the environment 400 capturedin the raw image data 526. The calibration module 532 may compare thekeypoints identified from the raw image data 526 to correspondingkeypoints in the filtered surfel map 132 to identify differences betweenthe expected locations of the flagged features and the actual locationsfor the flagged features. For example, the calibration module 532 maycompare vectors generated from raw image data 526 to correspondingportions in the filtered surfel map 132 to identify the distancesbetween an expected keypoint location and an actual keypoint locationfor each of the keypoints. The calibration module 532 can use thesedifferences to generate the updated camera parameters 538 b.

In some cases, updating the camera parameters includes adjusting one ormore camera settings or characteristics. For example, the output of thecalibration module 532 may indicate that the one or more of the camera'saperture, shutter speed, ISO, radial distortion parameters for a lens ofthe camera, and/or a focal length of the camera (e.g., to improveaccuracy of images captured using the camera).

FIG. 6 is a flow diagram of an example process for using an environmentmap to calibrate one or more sensors of a system. The process can beperformed, at least in part, using the on-board system 110 describedherein with respect to FIG. 1 . The example process will be described asbeing performed by a system of one or more computers.

The system obtains a three-dimensional representation of a real-worldenvironment comprising a plurality of surfels (602). With respect toFIG. 4 , the real-world environment can be the environment 400. Thethree-dimensional representation of the real-world environment can be asurfel map. For example, with respect to FIG. 1 , the three-dimensionalrepresentation of the real-world environment can be the global surfelmap 145 or another global surfel map for the environment 400. The globalsurfel map can be an existing surfel map that has been generated bycombining sensor data captured by multiple vehicles navigating throughthe real-world environment. The portion of the global surfel map cancorrespond to the environment 400, e.g., previously generated bycombining sensor data captured by one or more vehicles had navigatedthrough the environment 400. As an example, this global surfel map caninclude an indication of the road 404, the sidewalk 420, the tree 422,the bushes 424, the flagpole 426, the flag 428, and the sign 430.

In some cases, each of the surfels of the plurality of surfelscorresponds to a respective point of plurality of points in athree-dimensional space of the real-world environment. For example, withrespect to FIG. 5A, each of the surfels of the plurality of surfels inthe global surfel map 145 can have spatial coordinates, e.g., (x,y,z)defining a particular position of the respective surfel in athree-dimensional coordinate system of the environment 400 shown in FIG.4 . Additionally, or alternatively, each of the surfels of the pluralityof surfels in the global surfel map 145 can have orientationcoordinates, e.g., (pitch, yaw, roll) defining a particular orientationof the surface of the respective surfel. As another example, withrespect to FIG. 5A, each of the surfels of the plurality of surfels inthe global surfel map 145 can have spatial coordinates that define theparticular position of the respective surfel in a three-dimensionalcoordinate system (e.g., of the environment 400 shown in FIG. 4 ) and anormal vector, e.g. a vector with a magnitude of 1, that defines theorientation of the surface of the respective surfel at the particularposition.

The global surfel map 145 depicts the environment 400 using multiplesurfels. Each of the surfels can have one or more labels andcorresponding confidences. The labels can, for example, identify theobject that the surfel is conveying, identify a material that the objectis made of, identify a permanence of the object, identify a color of theobject (or a portion of the object), identify an opaqueness of theobject (or a portion of the object), etc. With respect to FIG. 5A, thelabels can correspond to particular surfel classifications which aredepicted, for example, using different colors/fill densities.

Multiple surfels of the plurality of surfels can be associated with aplurality of different classifications. For example, each surfel in theglobal surfel map 145 may be associated with one of the five surfelclassification shown in the surfel classifications key 502 a.

The system selects one or more surfels of the representation having aparticular classification (604). For example, the environment predictionsystem 130 (or the server system 122) may select all surfels in theglobal surfel map 145 that are associated with the third surfelclassification (e.g., grass/bush/tree) and the fourth surfelclassification (e.g., flag/cloth/clothing). The surfels and/or theclassification of surfels selected may be based on the type ofenvironment that vehicle is traveling in.

In some cases, selecting the one or more surfels of the surfelrepresentation having the particular classification includes selectingall surfels in the surfel representation that are associated withvegetation. For example, the environment prediction system 130 (or theserver system 122) may select all surfels in the global surfel map 145that are associated with the third surfel classification (e.g.,grass/bush/tree) based on the tendency of vegetation to be unreliable.Surfels that represent vegetation may be considered unreliable due tofrequent changes in the position, size, and/or shape of the vegetation,and/or due to vegetation having too fine of detail (e.g., leaves) toconsistently perceive from sensor data collected using one or more ofthe sensors of the sensor subsystems 120.

In some cases, selecting the one more surfels of the surfelrepresentation having the particular classification includes selectingall surfels in the surfel representation that are associated with typesof objects other than one or more of the following types of objects:buildings; roadways; road signs; or road barriers. For example, theenvironment prediction system 130 may select all surfels in the globalsurfel map that are not associated with the first surfel classification(e.g., sign/metal) and the second surfel classification (e.g., roadway).

The system can optionally update the representation based on the one ormore surfels (606). For example, the environment prediction system 130(or the server system 122) may filter out the selected surfels. That is,the environment prediction system 130 may remove or hide any surfels inthe global surfel map 145 that were associated with the third surfelclassification (e.g., grass/bush/tree) and the fourth surfelclassification (e.g., flag/cloth/clothing). In updating therepresentation based on the one or more surfels, the system may generatea filtered surfel map. For example, the environment prediction system130 may generate the filtered surfel map 132 from the global surfel map145 by removing or hiding any surfels in the global surfel map 145 thatwere associated with the third surfel classification and the fourthsurfel classification.

In some cases, selecting the one or more surfels of the representationhaving the particular classification includes the system selecting oneor more surfels in the representation that have been associated with aparticular object of the real-world environment; and updating therepresentation based on the one or more surfels includes the systemfiltering the three-dimensional representation to remove or hide allsurfels in the three-dimensional representation that have beenassociated with a particular object of the real-world environment. Forexample, the environment prediction system 130 may select surfels thathave been labelled as being a bush, a tree, a traffic cone, etc. Ingenerating the filtered surfel map 132, the environment predictionsystem 130 may remove or hide the selected surfels associated with bushobjects, tree objects, and/or traffic cone objects.

In some cases, selecting the one or more surfels of the representationhaving the particular classification includes the system selecting theone or more surfels in the representation that have the particularclassification based on a type of environment of the real-worldenvironment; and updating the representation based on the one or moresurfels includes the system filtering the three-dimensionalrepresentation to remove or hide the one or more surfels. For example,based on the environment 400 being a four-lane road, the environmentprediction system 130 may determine that surfels associated with thethird surfel classification (e.g., grass/bush/tree) should be removed orhidden based on a lookup table that correlates four-lane roads with theremoval/hiding of surfels associated with the third surfelclassification.

In some cases, selecting the one or more surfels of the representationhaving the particular classification includes the system selecting oneor more surfels in the representation that have been identified asuntrustworthy; and updating the representation based on the one or moresurfels includes the system filtering the three-dimensionalrepresentation to remove or hide the one or more surfels in therepresentation that have been identified as untrustworthy. For example,the environment prediction system 130 may identify surfels in the globalsurfel map that do not have a threshold level of confidence (e.g., inthe object they represent, type of object that they represent, level ofpermanence, etc.). The environment prediction system 130 may proceed toremove or hide these identified surfels in generating the filteredsurfel map 132.

In some cases, the system does not update the representation. Forexample, instead of filtering surfels, the filter module 530 identifiesreliable and/or unreliable surfels. That is, the filter module 530 may,using the techniques described above, identify a subset of surfels inthe global surfel map 145 that are sufficiently reliable to use forcalibration. For example, the filter module 530 may identify all surfelsassociated with the first surfel classification (e.g., sign/metal) asbeing sufficiently reliable to use to calibrate one or more on-boardsensors of the vehicle 102's on-board system 110. The filter module 530may optionally tag the surfels in the subset of surfels to indicate thatthey are sufficiently reliable to use for calibration. A calibrationmodule 532, described below with respect to FIGS. 5C-5D, may identifythe subset of surfels using the tags and proceed to use only thosesurfels for sensor calibration.

In some cases, updating the representation includes tagging one or moresurfels to indicate that the one or more surfels are sufficientlyreliable for calibration. For example, the environment prediction system130 may attach a tag to each surfel in a subset of surfels of the globalsurfel map 145 that the environment prediction system 130 identified asbeing sufficiently reliable for calibration. Similarly, in some cases,updating the representation includes tagging one or more surfels toindicate that the one or more surfels are not sufficiently reliable forcalibration. For example, the environment prediction system 130 mayattach a tag to each surfel in a subset of surfels of the global surfelmap 145 that the environment prediction system 130 identified as beingnot sufficiently reliable for calibration.

The system receives input sensor data from one or more sensors installedon the autonomous or semi-autonomous vehicle (608). The input sensordata can include electromagnetic radiation. As an example, the inputsensor data can include data collected by one or more of lidar systemsthat detect reflections of laser light, radar systems that detectreflections of radio waves, or camera systems that detect reflections ofvisible light. With respect to FIG. 1 , the input sensor data can be theraw sensor measurements or the raw sensor data 125 compiled by thesensor subsystems 120. With respect to FIG. 1 and FIG. 4 , theautonomous vehicle can be the vehicle 102. The on-board system 110 caninclude the sensors that collect the input sensor data. For example, theon-board system 110 can include one or more of a lidar system thatdetect reflections of laser light, a radar system that detectreflections of radio waves, or a camera system that detect reflectionsof visible light. The sensor data generated by a given sensor, e.g., ofthe on-board system 110, generally indicates a distance, a direction,and an intensity of reflected radiation. For example, a sensor of theon-board system 110 can transmit one or more pulses of electromagneticradiation in a particular direction and can measure the intensity of anyreflections as well as the time that the reflection was received. Adistance can be computed by determining how long it took between a pulseand its corresponding reflection. The sensor can continually sweep aparticular space in angle, azimuth, or both. Sweeping in azimuth, forexample, can allow a sensor to detect multiple objects along the sameline of sight.

The system compares the input sensor data to the updated to identifydifferences between the observation and the updated representation(610). The updated representation may also be a three-dimensionalrepresentation, e.g., of a particular environment that the vehicle istraveling in or is about to enter. In general, this can includeidentifying multiple points in the input sensor data, and performingpoint matching to find corresponding points in the updatedthree-dimensional representation. The locations of the matching pointsmay be compared to an expected location of the matching points to makecalibration adjustments. For example, as described in detail above withrespect to FIG. 5C, the environment prediction system 130 may compareraw lidar sensor data 524 to the filtered surfel map 132 by performinglidar dense calibration. Similarly, as described in detail above withrespect to FIG. 5D, the environment prediction system 130 may compareraw image data 526 to the filtered surfel map 132 by performing camerafeature matching.

In some cases, comparing the input sensor data to the updatedrepresentation to identify differences between the observation and therepresentation includes the system: estimating three-dimensional pointdata from the laser data, where the observation is the three-dimensionalpoint data; and performing dense image matching using thethree-dimensional point data to identify point matches between theobservation and the three-dimensional representation. For example, thecalibration module 532 may generate a point cloud from the raw lidarsensor data 524. The calibration module 532 may proceed to compare thepoint cloud (or a portion of the point cloud) to the filtered surfel map132 to identify point matches between the point cloud (or the portion ofthe point cloud) and the filtered surfel map 132. The differencesbetween the observation and the three-dimensional representation caninclude distances between the matching points in the point data andcorresponding surfels in the representation, and calibrating the atleast one sensor can include the system calibrating the lidar systemusing the distances. For example, the calibration module 532 maycalculate distances between the matching points in the point data andthe corresponding surfels in the representation (e.g., the expectedlocations of the point matches). The calibration module 532 may proceedto generate the updated lidar sensor parameters 534 b using thesedistances.

In some cases, comparing the input sensor data to the three-dimensionalrepresentation to identify differences between the observation and thethree-dimensional representation includes the system performing featurematching using the image data and the three-dimensional representationto identify features in the image data that match corresponding featuresin the three-dimensional representation. For example, the calibrationmodule 532 of the environment prediction system 130 may perform featurematching using the raw image data 526 (or vectors derived from the rawimage data 526) and the filtered surfel map 132. In performing thefeature matching, the calibration module 532 can match features in theimages to corresponding features in the filtered surfel map 132.Featuring matching may include, for example, identifying keypoints inthe raw image data 526 and matching those keypoints to correspondingpoints in the filtered surfel map 132.

The features in the three-dimensional representation may correspond togroupings of surfels in the three-dimensional representation. Forexample, the flagged features shown in FIG. 5C are those groupings ofsurfels that have been associated with the first surfel classification(e.g., sign/metal). This may include a first grouping of surfelscorresponding to the flagpole 426 and a second grouping of surfelscorresponding to the sign 430.

The differences between the observation and the three-dimensionalrepresentation may include distances between the features in the imagedata and corresponding groupings of surfels in the three-dimensionalrepresentation, and calibrating the at least one sensor includes thesystem calibrating the one or more cameras using the distances. Forexample, the calibration module 532 may calculate distances between anexpected location for the location of the sign 430 and matched locationof the sign 430 with respect to the filtered surfel map 432. Thecalibration module 532 may use these distances to generate the updatedcamera parameters 538 b.

The system calibrates at least one sensor of the one or more sensorsusing the differences (612). In general, the system may use thedistances between locations of point matches to expected locations ofpoint matches to determine updates to make to the parameters of one ormore sensors, such as a camera and/or a lidar sensor. As an example,with respect to FIG. 5C, the calibration module 532 of the environmentprediction system 130 may update a laser range offset, various positionoffsets, and/or various rotation offsets for the lidar sensor based ondifference between actual locations of matched points from a point cloudgenerated using the raw lidar sensor data 524 and expected locations.

In some cases, calibrating the one or more cameras comprises adjustingone or more of the following: an offset of a camera of the one or morecameras; a rotation a camera of the one or more cameras; a radialdistortion parameter of a lens of a camera of the one or more cameras;or a focal length of a lens of a camera of the one or more cameras.

In some cases, the system determines a type of environment of thereal-world environment using the representation. For example, theenvironment prediction system 130 may use the global surfel map 145 toidentify a type of environment that the environment 400. Specifically,the environment prediction system 130 may look at theclassifications/labels of the surfels that are used to represent theenvironment (e.g., if more than 50% of the surfels correspond to thefifth surfel classification for concrete, the environment predictionsystem 130 may conclude that the type of environment is a city). Theenvironment prediction system 130 may also use the global surfel map 145to identify a type of environment based on the types of objects in theenvironment 400 and/or based on the size of objects in the environment400 (e.g., width of the road 404). Calibrating the at least one sensorcan include the system determining that calibration is viable based onthe type of environment. For example, the calibration module 532 of theenvironment prediction system 130 may determine that not calibrationshould be attempted due the environment being classified as a heavilyforested area (e.g., a type of environment that includes objects thatare frequently changing in appearance, that are frequently changingposition, that include too fine of detail to accurately use forcalibration, etc.). The calibration module 532 may similarly determinethat calibration is not viable based on, for example, current weatherconditions in the environment 400.

In some cases, determining the type of environment of the real-worldenvironment using the representation includes the system: analyzing therepresentation or a portion of the representation to identifyclassifications of surfels in the representation or the portion of therepresentation; and based on percentages of the classifications of thesurfels in the representation or the portion of the representation,determining a type of environment from multiple types of environments.For example, the environment prediction system 130 may determine thatthe environment 400 is a heavily forested if at least 82% of the surfelsthat represent the environment 400 in the global surfel map 145 areclassified as vegetation.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively, or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, off-the-shelf or custom-made parallel processingsubsystems, e.g., a GPU or another kind of special-purpose processingsubsystem. The apparatus can also be, or further include, specialpurpose logic circuitry, e.g., an FPGA (field programmable gate array)or an ASIC (application-specific integrated circuit). The apparatus canoptionally include, in addition to hardware, code that creates anexecution environment for computer programs, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, anoperating system, or a combination of one or more of them.

A computer program which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub-programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

For a system of one or more computers to be configured to performparticular operations or actions means that the system has installed onits software, firmware, hardware, or a combination of them that inoperation cause the system to perform the operations or actions. For oneor more computer programs to be configured to perform particularoperations or actions means that the one or more programs includeinstructions that, when executed by data processing apparatus, cause theapparatus to perform the operations or actions.

As used in this specification, an “engine,” or “software engine,” refersto a software implemented input/output system that provides an outputthat is different from the input. An engine can be an encoded block offunctionality, such as a library, a platform, a software development kit(“SDK”), or an object. Each engine can be implemented on any appropriatetype of computing device, e.g., servers, mobile phones, tabletcomputers, notebook computers, music players, e-book readers, laptop ordesktop computers, PDAs, smart phones, or other stationary or portabledevices, that includes one or more processors and computer readablemedia. Additionally, two or more of the engines may be implemented onthe same computing device, or on different computing devices.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and pointing device, e.g., a mouse, trackball, or a presencesensitive display or other surface by which the user can provide inputto the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input. Inaddition, a computer can interact with a user by sending documents toand receiving documents from a device that is used by the user; forexample, by sending web pages to a web browser on a user's device inresponse to requests received from the web browser. Also, a computer caninteract with a user by sending text messages or other forms of messageto a personal device, e.g., a smartphone, running a messagingapplication, and receiving responsive messages from the user in return.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

In addition to the embodiments described above, the followingembodiments are also innovative:

Embodiment 1 is a method comprising:

-   -   obtaining a three-dimensional surfel representation of a        real-world environment, the surfel representation comprising a        plurality of surfels,        -   wherein each of the surfels corresponds to a respective            location of plurality of locations in a three-dimensional            space of the real-world environment, and        -   wherein multiple surfels of the plurality of surfels are            associated with a plurality of different classifications;    -   selecting one or more surfels of the surfel representation        having a particular classification of the different        classifications;    -   receiving input sensor data from one or more sensors installed        on an autonomous or semi-autonomous vehicle, the input sensor        data indicating an observation of at least a portion of the        real-world environment;    -   comparing the input sensor data to the surfel representation to        identify one or more differences between the observation and the        surfel representation; and    -   calibrating at least one sensor of the one or more sensors using        the one or more differences between the observation and the        surfel representation.

Embodiment 2 is the method of embodiment 1, comprising updating thesurfel representation based on the one or more surfels,

-   -   wherein comparing the input sensor data to the surfel        representation to identify the one or more differences between        the observation and the surfel representation comprises        comparing the input sensor data to the updated surfel        representation to identify one or more differences between the        observation and the updated surfel representation.

Embodiment 3 is the method of embodiments 2, wherein updating the surfelrepresentation based on the one or more surfels comprises filtering thesurfel representation to remove or hide the one or more surfels havingthe particular classification.

Embodiment 4 is the method of any one of embodiments 2-3, wherein:

-   -   selecting the one or more surfels of the surfel representation        having the particular classification comprises selecting all        surfels in the surfel representation that have the particular        classification; and    -   updating the surfel representation based on the one or more        surfels comprises filtering the surfel representation to remove        or hide all surfels in the representation that have the        particular classification.

Embodiment 5 is the method of any one of embodiments 2-4, wherein:

-   -   selecting the one or more surfels of the surfel representation        having the particular classification comprises selecting one or        more surfels in the surfel representation that have been        associated with a particular object of the real-world        environment; and    -   updating the surfel representation based on the one or more        surfels comprises filtering the surfel representation to remove        or hide all surfels in the surfel representation that have been        associated with a particular object of the real-world        environment.

Embodiment 6 is the method of any one of embodiments 2-5, wherein:

-   -   selecting the one or more surfels of the surfel representation        having the particular classification comprises selecting the one        or more surfels in the surfel representation that have the        particular classification based on a type of environment of the        real-world environment; and    -   updating the surfel representation based on the one or more        surfels comprises filtering the surfel representation to remove        or hide the one or more surfels.

Embodiment 7 is the method of any one of embodiments 2-6, wherein:

-   -   selecting the one or more surfels of the surfel representation        having the particular classification comprises selecting one or        more surfels in the surfel representation that have been        identified as untrustworthy; and    -   updating the surfel representation based on the one or more        surfels comprises filtering the surfel representation to remove        or hide the one or more surfels in the surfel representation        that have been identified as untrustworthy.

Embodiment 8 is the method of any one of embodiments 2-7, whereinupdating the surfel representation based on the one or more surfelscomprises attaching a tag to surfels in the surfel representation otherthan the one or more surfels, wherein the tag indicates that thecorresponding surfels are sufficiently reliable.

Embodiment 9 is the method of any one of embodiments 2-8, whereinupdating the surfel representation based on the one or more surfelscomprises attaching a tag to the one or more surfels, wherein the tagindicates that the corresponding surfels are not sufficiently reliable.

Embodiment 10 is the method of any one of embodiments 1-9, whereinselecting the one or more surfels of the surfel representation havingthe particular classification comprises selecting all surfels in thesurfel representation that are associated with vegetation.

Embodiment 11 is the method of any one of embodiments 1-10, whereinselecting the one more surfels of the surfel representation having theparticular classification comprises selecting all surfels in the surfelrepresentation that are associated with types of objects other than oneor more of the following types of objects:

-   -   buildings;    -   roadways;    -   road signs; or    -   road barriers.

Embodiment 12 is the method of any one of embodiments 1-11, wherein:

-   -   receiving input sensor data from the sensors installed on the        autonomous or semi-autonomous vehicle comprises receiving laser        data from a lidar system, and    -   the one or more sensors installed on the autonomous or        semi-autonomous vehicle include the lidar system.

Embodiment 13 is the method of any one of embodiments 1-12, wherein:

-   -   comparing the input sensor data to the surfel representation to        identify the one or more differences between the observation and        the surfel representation comprises:        -   estimating three-dimensional point data from the laser data,            wherein the observation is the three-dimensional point data;            and        -   performing dense image matching using the three-dimensional            point data to identify point matches between the observation            and the surfel representation,    -   the one or more differences between the observation and the        surfel representation include one or more distances between the        matching points in the point data and corresponding surfels in        the surfel representation, and    -   calibrating the at least one sensor comprises calibrating the        lidar system using the one or more distances.

Embodiment 14 is the method of any one of embodiments 1-13, wherein:

-   -   receiving input sensor data from the one or more sensors        installed on the autonomous or semi-autonomous vehicle comprises        receiving image data from one or more cameras, and    -   the one or more sensors installed on the autonomous or        semi-autonomous vehicle include the one or more cameras.

Embodiment 15 is the method of any one of embodiments 1-14, wherein:

-   -   comparing the input sensor data to the surfel representation to        identify the one or more differences between the observation and        the surfel representation comprises performing feature matching        using the image data and the surfel representation to identify        one or more features in the image data that match corresponding        one or more features in the surfel representation,    -   the one or more features in the surfel representation correspond        to one or more groupings of surfels in the surfel        representation,    -   the one or more differences between the observation and the        three-dimensional representation include one or more distances        between the one or more features in the image data and        corresponding one or more groupings of surfels in the surfel        representation, and    -   calibrating the at least one sensor comprises calibrating the        one or more cameras using the one or more distances.

Embodiment 16 is the method of any one of embodiments 1-15, whereincalibrating the one or more cameras comprises adjusting one or more ofthe following:

-   -   an offset of a camera of the one or more cameras;    -   a rotation a camera of the one or more cameras;    -   radial distortion parameters of a lens of a camera of the one or        more cameras; or    -   a focal length of a lens of a camera of the one or more cameras.

Embodiment 17 is the method of any one of embodiments 1-16, comprisingdetermining a type of environment of the real-world environment usingthe surfel representation,

-   -   wherein calibrating the at least one sensor comprises        determining that calibration is viable based on the type of        environment.

Embodiment 18 is the method of any one of embodiments 1-17, whereindetermining the type of environment of the real-world environment usingthe surfel representation comprises:

-   -   analyzing the surfel representation or a portion of the surfel        representation to identify classifications of surfels in the        surfel representation or the portion of the surfel        representation; and    -   based on percentages of the classifications of the surfels in        the surfel representation or the portion of the surfel        representation, determining a type of environment from multiple        types of environments.

Embodiment 19 is a system comprising: one or more computers and one ormore storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform the method of any one of embodiments 1 to 18.

Embodiment 20 is a computer storage medium encoded with a computerprogram, the program comprising instructions that are operable, whenexecuted by data processing apparatus, to cause the data processingapparatus to perform the method of any one of embodiments 1 to 18.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain some cases, multitasking and parallel processing maybe advantageous.

What is claimed is:
 1. A computer implemented method comprising:obtaining a three-dimensional surfel representation of a real-worldenvironment generated from initial sensor data captured by a pluralityof vehicles traveling in the real-world environment, the surfelrepresentation comprising a plurality of surfels, receiving input sensordata from one or more sensors installed on a first vehicle, the inputsensor data indicating an observation of the real-world environment;determining, from the input sensor data, one or more matching featuresin the sensor data that match corresponding features in the surfelrepresentation; computing, based on the one or more matching features,one or more differences between the sensor data and the surfelrepresentation; and calibrating at least one sensor of the one or moresensors using the one or more differences between the sensor data andthe surfel representation.
 2. The method of claim 1, comprising updatingthe surfel representation based on the one or more surfels, whereincomparing the input sensor data to the surfel representation to identifythe one or more differences between the observation and the surfelrepresentation comprises comparing the input sensor data to the updatedsurfel representation to identify one or more differences between theobservation and the updated surfel representation.
 3. The method ofclaim 2, wherein updating the surfel representation based on the one ormore surfels comprises filtering the surfel representation to remove orhide one or more surfels having a particular classification.
 4. Themethod of claim 2, wherein updating the surfel representation based onthe one or more surfels comprises filtering the surfel representation toremove or hide all surfels in the representation that have a particularclassification.
 5. The method of claim 2, wherein updating the surfelrepresentation based on the one or more surfels comprises filtering thesurfel representation to remove or hide all surfels in the surfelrepresentation that have been associated with a particular object of thereal-world environment.
 6. The method of claim 2, wherein updating thesurfel representation based on the one or more surfels comprisesfiltering the surfel representation to remove or hide the one or moresurfels.
 7. The method of claim 2, wherein updating the surfelrepresentation based on the one or more surfels comprises filtering thesurfel representation to remove or hide the one or more surfels in thesurfel representation that have been identified as untrustworthy.
 8. Themethod of claim 2, wherein updating the surfel representation based onthe one or more surfels comprises attaching a tag to surfels in thesurfel representation other than the one or more surfels, wherein thetag indicates that the corresponding surfels are sufficiently reliable.9. The method of claim 2, wherein updating the surfel representationbased on the one or more surfels comprises attaching a tag to the one ormore surfels, wherein the tag indicates that the corresponding surfelsare not sufficiently reliable.
 10. The method of claim 1, whereinselecting the one or more surfels of the surfel representation comprisesselecting all surfels in the surfel representation that are associatedwith vegetation.
 11. The method of claim 1, wherein the one or morematching features comprise one or more of the following types ofobjects: buildings; roadways; road signs; or road barriers.
 12. Themethod of claim 1, wherein: receiving input sensor data from the sensorsinstalled on the first vehicle comprises receiving laser data from alidar system, and the one or more sensors installed on the first vehicleinclude the lidar system.
 13. The method of claim 12, wherein: comparingthe input sensor data to the surfel representation to identify the oneor more differences between the observation and the surfelrepresentation comprises: estimating three-dimensional point data fromthe laser data, wherein the observation is the three-dimensional pointdata; and performing dense image matching using the three-dimensionalpoint data to identify point matches between the observation and thesurfel representation, the one or more differences between theobservation and the surfel representation include one or more distancesbetween the matching points in the point data and corresponding surfelsin the surfel representation, and calibrating the at least one sensorcomprises calibrating the lidar system using the one or more distances.14. The method of claim 1, wherein: receiving input sensor data from theone or more sensors installed on the autonomous or semi-autonomousvehicle comprises receiving image data from one or more cameras, and theone or more sensors installed on the autonomous or semi-autonomousvehicle include the one or more cameras.
 15. The method of claim 14,wherein: comparing the input sensor data to the surfel representation toidentify the one or more differences between the observation and thesurfel representation comprises performing feature matching using theimage data and the surfel representation to identify one or morefeatures in the image data that match corresponding one or more featuresin the surfel representation, the one or more features in the surfelrepresentation correspond to one or more groupings of surfels in thesurfel representation, the one or more differences between theobservation and the three-dimensional representation include one or moredistances between the one or more features in the image data andcorresponding one or more groupings of surfels in the surfelrepresentation, and calibrating the at least one sensor comprisescalibrating the one or more cameras using the one or more distances. 16.The method of claim 15, wherein calibrating the one or more camerascomprises adjusting one or more of the following: an offset of a cameraof the one or more cameras; a rotation a camera of the one or morecameras; radial distortion parameters of a lens of a camera of the oneor more cameras; or a focal length of a lens of a camera of the one ormore cameras.
 17. The method of claim 1, comprising determining a typeof environment of the real-world environment using the surfelrepresentation, wherein calibrating the at least one sensor comprisesdetermining that calibration is viable based on the type of environment.18. The method of claim 17, wherein determining the type of environmentof the real-world environment using the surfel representation comprises:analyzing the surfel representation or a portion of the surfelrepresentation to identify classifications of surfels in the surfelrepresentation or the portion of the surfel representation; and based onpercentages of the classifications of the surfels in the surfelrepresentation or the portion of the surfel representation, determininga type of environment from multiple types of environments.
 19. A systemcomprising: one or more computers; and one or more storage devicesstoring instructions that are operable, when executed by the one or morecomputers, to cause the one or more computers to perform operationscomprising: obtaining a three-dimensional surfel representation of areal-world environment generated from initial sensor data captured by aplurality of vehicles traveling in the real-world environment, thesurfel representation comprising a plurality of surfels, receiving inputsensor data from one or more sensors installed on a first vehicle, theinput sensor data indicating an observation of the real-worldenvironment; determining, from the input sensor data, one or morematching features in the sensor data that match corresponding one ormore features in the surfel representation; computing, based on the oneor more matching features, one or more differences between the sensordata and the surfel representation; and calibrating at least one sensorof the one or more sensors using the one or more differences between thesensor data and the surfel representation.
 20. A non-transitory computerstorage medium encoded with a computer program, the program comprisinginstructions that are operable, when executed by data processingapparatus, to cause the data processing apparatus to perform operationscomprising: obtaining a three-dimensional surfel representation of areal-world environment generated from initial sensor data captured by aplurality of vehicles traveling in the real-world environment, thesurfel representation comprising a plurality of surfels, receiving inputsensor data from one or more sensors installed on a first vehicle, theinput sensor data indicating an observation of the real-worldenvironment; determining, from the input sensor data, one or morematching features in the surfel representation; computing, based on theone or more matching features, one or more differences between thesensor data and the surfel representation; and calibrating at least onesensor of the one or more sensors using the one or more differencesbetween the sensor data and the surfel representation.